[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:48:59 dist-test-slave-jrck krb5kdc[3082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:48:59 dist-test-slave-jrck krb5kdc[3082](info): set up 2 sockets
Sep 28 21:48:59 dist-test-slave-jrck krb5kdc[3082](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 21:49:00.300344 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:00 dist-test-slave-jrck krb5kdc[3082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:00.409492 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:36557
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:36557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:00.487913 3095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:00.488125 3095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:00.488155 3095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:00.489337 3095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:00.489378 3095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:00.489405 3095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:00.489420 3095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:00.489436 3095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:00.490914 3095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:36557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:36557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:00.491163 3095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:00.491393 3095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:49:00.496390 3095 server_base.cc:1034] running on GCE node
W20260928 21:49:00.496573 3100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:00.496578 3101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:00.496563 3103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:00.496946 3095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:00.497205 3095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:00.498344 3095 hybrid_clock.cc:648] HybridClock initialized: now 1790632140498324 us; error 37 us; skew 500 ppm
Sep 28 21:49:00 dist-test-slave-jrck krb5kdc[3082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632140, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:00.500662 3095 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:49:00.501070 3095 webserver.cc:460] Webserver started at http://127.3.0.254:38589/ using document root <none> and password file <none>
I20260928 21:49:00.501273 3095 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:00.501317 3095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:00.501437 3095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:00.502450 3095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "7b7b55ea0a8b4bf481942f859d3902d0"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "53cc5a402d015fcbc333f050a710f84e"
server_key_iv: "021ea1a708725c7090d14eca307fedf0"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.502759 3095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "7b7b55ea0a8b4bf481942f859d3902d0"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "53cc5a402d015fcbc333f050a710f84e"
server_key_iv: "021ea1a708725c7090d14eca307fedf0"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.503965 3095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:49:00.504593 3110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:00.504739 3095 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:00.504814 3095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "7b7b55ea0a8b4bf481942f859d3902d0"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "53cc5a402d015fcbc333f050a710f84e"
server_key_iv: "021ea1a708725c7090d14eca307fedf0"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.504880 3095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:00.536881 3095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:00.538847 3095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:00.538996 3095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:00.542791 3095 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:36557
I20260928 21:49:00.542826 3162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:36557 every 8 connection(s)
I20260928 21:49:00.543164 3095 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:00.543947 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 3095
I20260928 21:49:00.544055 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:00.544104 3163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:00.544867 3075 external_mini_cluster.cc:1428] Setting key 79e6706a072b75e1e919da7a8d3ad264
I20260928 21:49:00.547165 3163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0: Bootstrap starting.
I20260928 21:49:00.547868 3163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:00.548170 3163 log.cc:824] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:00.548995 3163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0: No bootstrap required, opened a new log
I20260928 21:49:00.551366 3163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7b55ea0a8b4bf481942f859d3902d0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36557 } }
I20260928 21:49:00.551501 3163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:00.551522 3163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7b55ea0a8b4bf481942f859d3902d0, State: Initialized, Role: FOLLOWER
I20260928 21:49:00.551635 3163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7b55ea0a8b4bf481942f859d3902d0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36557 } }
I20260928 21:49:00.551712 3163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:00.551736 3163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:00.551781 3163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:00.552320 3163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7b55ea0a8b4bf481942f859d3902d0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36557 } }
I20260928 21:49:00.552429 3163 leader_election.cc:302] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b7b55ea0a8b4bf481942f859d3902d0; no voters:
I20260928 21:49:00.552618 3163 leader_election.cc:288] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:00.552649 3168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:00.552788 3168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [term 1 LEADER]: Becoming Leader. State: Replica: 7b7b55ea0a8b4bf481942f859d3902d0, State: Running, Role: LEADER
I20260928 21:49:00.552876 3163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:00.552898 3168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7b55ea0a8b4bf481942f859d3902d0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36557 } }
I20260928 21:49:00.553243 3170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b7b55ea0a8b4bf481942f859d3902d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7b55ea0a8b4bf481942f859d3902d0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36557 } } }
I20260928 21:49:00.553236 3168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b7b55ea0a8b4bf481942f859d3902d0. Latest consensus state: current_term: 1 leader_uuid: "7b7b55ea0a8b4bf481942f859d3902d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7b55ea0a8b4bf481942f859d3902d0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36557 } } }
I20260928 21:49:00.553335 3170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:00.553340 3168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:00.553586 3175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:00.554087 3175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:00.555692 3175 catalog_manager.cc:1347] Generated new cluster ID: 2ab5056ebd104653b661c0a6c355c89d
I20260928 21:49:00.555745 3175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:49:00 dist-test-slave-jrck krb5kdc[3082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:00.562079 3175 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:00.562672 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:00.545767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47192 (local address 127.3.0.254:36557)
0928 21:49:00.546056 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:00.546063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:00.546150 (+ 87us) server_negotiation.cc:407] Connection header received
0928 21:49:00.548927 (+ 2777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:00.548934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:00.548984 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:00.549119 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:00.551222 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:00.552277 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:00.553794 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:00.554437 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:00.554522 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:00.558682 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:00.558687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:00.558687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:00.558700 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:00.560688 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:00.561166 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:00.561169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:00.561170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:00.561309 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:00.561580 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:00.561580 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:00.561581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:00.561872 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:00.562025 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:00.562343 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 21:49:00.562470 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":74,"threads_started":1}
I20260928 21:49:00.562752 3175 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:00.567628 3175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b7b55ea0a8b4bf481942f859d3902d0: Generated new TSK 0
I20260928 21:49:00.567804 3175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:49:00.595371 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36557
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:00.675148 3189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:00.675367 3189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:00.675401 3189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:00.676584 3189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:00.676627 3189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:00.676664 3189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:00.678124 3189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36557
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:00.678363 3189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:00.678568 3189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:00.683703 3195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:00.683789 3189 server_base.cc:1034] running on GCE node
W20260928 21:49:00.683708 3197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:00.683877 3194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:00.684114 3189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:00.684377 3189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:00.685539 3189 hybrid_clock.cc:648] HybridClock initialized: now 1790632140685536 us; error 40 us; skew 500 ppm
Sep 28 21:49:00 dist-test-slave-jrck krb5kdc[3082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632140, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:00.687938 3189 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:49:00.688351 3189 webserver.cc:460] Webserver started at http://127.3.0.193:43531/ using document root <none> and password file <none>
I20260928 21:49:00.688575 3189 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:00.688620 3189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:00.688714 3189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:00.689723 3189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "fac080b7bd5c4753a53dafcffcf995db"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "32440ad8f690b376ab3a91180661bb18"
server_key_iv: "a2bec4487aae4e272f34fccbafe3f54c"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.690062 3189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "fac080b7bd5c4753a53dafcffcf995db"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "32440ad8f690b376ab3a91180661bb18"
server_key_iv: "a2bec4487aae4e272f34fccbafe3f54c"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.691231 3189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:00.691862 3204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:00.692001 3189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:00.692081 3189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "fac080b7bd5c4753a53dafcffcf995db"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "32440ad8f690b376ab3a91180661bb18"
server_key_iv: "a2bec4487aae4e272f34fccbafe3f54c"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.692137 3189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:00.701512 3189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:00.703284 3189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:00.703393 3189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:00.703583 3189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:00.703835 3189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:00.703864 3189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:00.703884 3189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:00.703900 3189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:00.709331 3189 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:39279
I20260928 21:49:00.709400 3317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:39279 every 8 connection(s)
I20260928 21:49:00.709678 3189 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:49:00.710352 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 3189
I20260928 21:49:00.710443 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:00.710599 3075 external_mini_cluster.cc:1428] Setting key 186e20f2dcba995c8110bb322c4b9132
Sep 28 21:49:00 dist-test-slave-jrck krb5kdc[3082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632140, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
I20260928 21:49:00.719738 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:00.710382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:54983 (local address 127.3.0.254:36557)
0928 21:49:00.710445 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:00.710448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:00.710629 (+ 181us) server_negotiation.cc:407] Connection header received
0928 21:49:00.711225 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:00.711226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:00.711236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:00.711289 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:00.712491 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:00.713236 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:00.714551 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:00.714818 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:00.714871 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:00.717293 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:00.717295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:00.717295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:00.717301 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:00.718574 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:00.718940 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:00.718941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:00.718941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:00.718989 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:00.719186 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:00.719186 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:00.719187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:00.719315 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:00.719375 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:00.719600 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 21:49:00.719668 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:49:00.719910 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:00.710340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36557 (local address 127.3.0.193:54983)
0928 21:49:00.710491 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:00.710541 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:00.711126 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:00.711368 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:00.711372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:00.712101 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:00.712373 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:00.712380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:00.713326 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:00.713327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:00.714460 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:00.714463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:00.714897 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:00.714898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:00.715042 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:00.715554 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:00.715564 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:00.717200 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:00.718644 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:00.718651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:00.718654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:00.718882 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:00.719047 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:00.719047 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:00.719048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:00.719142 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:00.719369 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:00.719373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:00.719495 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 21:49:00.719578 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 21:49:00.719698 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260928 21:49:00.720412 3318 heartbeater.cc:347] Connected to a master server at 127.3.0.254:36557
I20260928 21:49:00.720532 3318 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
I20260928 21:49:00.720719 3318 heartbeater.cc:511] Master 127.3.0.254:36557 requested a full tablet report, sending...
I20260928 21:49:00.721256 3127 ts_manager.cc:194] Registered new tserver with Master: fac080b7bd5c4753a53dafcffcf995db (127.3.0.193:39279)
I20260928 21:49:00.721858 3127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:54983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:49:00.732869 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36557
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:00.813237 3323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:00.813445 3323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:00.813468 3323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:00.814594 3323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:00.814630 3323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:00.814656 3323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:00.816061 3323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36557
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:00.816262 3323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:00.816448 3323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:00.821547 3329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:00.821692 3323 server_base.cc:1034] running on GCE node
W20260928 21:49:00.821560 3328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:00.821583 3331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:00.821995 3323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:00.822245 3323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:00.823392 3323 hybrid_clock.cc:648] HybridClock initialized: now 1790632140823380 us; error 25 us; skew 500 ppm
Sep 28 21:49:00 dist-test-slave-jrck krb5kdc[3082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632140, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:00.825703 3323 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:49:00.826084 3323 webserver.cc:460] Webserver started at http://127.3.0.194:38235/ using document root <none> and password file <none>
I20260928 21:49:00.826270 3323 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:00.826313 3323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:00.826423 3323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:00.827467 3323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "73d5b76d54324f7bb7f6bd1d9dd25a90"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "c7c4ce02eddd4fedb8ec8d42840d3aa4"
server_key_iv: "6314243f3f5ffa9d8a84e8f9c187b11f"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.827761 3323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "73d5b76d54324f7bb7f6bd1d9dd25a90"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "c7c4ce02eddd4fedb8ec8d42840d3aa4"
server_key_iv: "6314243f3f5ffa9d8a84e8f9c187b11f"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.829030 3323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:00.829684 3338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:00.829857 3323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:00.829937 3323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "73d5b76d54324f7bb7f6bd1d9dd25a90"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "c7c4ce02eddd4fedb8ec8d42840d3aa4"
server_key_iv: "6314243f3f5ffa9d8a84e8f9c187b11f"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.829990 3323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:00.846413 3323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:00.848641 3323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:00.848758 3323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:00.848950 3323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:00.849232 3323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:00.849264 3323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:00.849296 3323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:00.849319 3323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:00.855500 3323 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:36615
I20260928 21:49:00.855548 3451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:36615 every 8 connection(s)
I20260928 21:49:00.855859 3323 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:49:00.857021 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 3323
I20260928 21:49:00.857103 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:00.857247 3075 external_mini_cluster.cc:1428] Setting key edeee428c7f765c792c6a768ae27108e
Sep 28 21:49:00 dist-test-slave-jrck krb5kdc[3082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632140, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
I20260928 21:49:00.865669 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:00.856584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:35723 (local address 127.3.0.254:36557)
0928 21:49:00.856639 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:00.856641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:00.856814 (+ 173us) server_negotiation.cc:407] Connection header received
0928 21:49:00.857368 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:00.857370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:00.857383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:00.857446 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:00.858494 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:00.859173 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:00.860487 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:00.860766 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:00.860883 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:00.863002 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:00.863004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:00.863004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:00.863013 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:00.864422 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:00.864826 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:00.864827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:00.864827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:00.864893 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:00.865097 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:00.865099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:00.865099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:00.865236 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:00.865300 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:00.865535 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 21:49:00.865601 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:49:00.865825 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:00.856532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36557 (local address 127.3.0.194:35723)
0928 21:49:00.856691 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:00.856729 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:00.857235 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:00.857526 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:00.857530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:00.858183 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:00.858393 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:00.858399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:00.859261 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:00.859262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:00.860375 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:00.860378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:00.860839 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:00.860840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:00.860951 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:00.861434 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:00.861442 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:00.862904 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:00.864546 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:00.864551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:00.864553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:00.864754 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:00.864954 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:00.864955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:00.864955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:00.865043 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:00.865304 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:00.865308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:00.865427 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 21:49:00.865507 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:49:00.865626 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260928 21:49:00.866288 3452 heartbeater.cc:347] Connected to a master server at 127.3.0.254:36557
I20260928 21:49:00.866401 3452 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:00.866577 3452 heartbeater.cc:511] Master 127.3.0.254:36557 requested a full tablet report, sending...
I20260928 21:49:00.866955 3127 ts_manager.cc:194] Registered new tserver with Master: 73d5b76d54324f7bb7f6bd1d9dd25a90 (127.3.0.194:36615)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
I20260928 21:49:00.867352 3127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:35723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:49:00.880448 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36557
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:00.959268 3457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:00.959462 3457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:00.959484 3457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:00.960606 3457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:00.960641 3457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:00.960666 3457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:00.962054 3457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36557
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:00.962236 3457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:00.962417 3457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:00.967027 3462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:00.967038 3463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:00.967141 3457 server_base.cc:1034] running on GCE node
W20260928 21:49:00.967038 3465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:00.967398 3457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:00.967633 3457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:00.968761 3457 hybrid_clock.cc:648] HybridClock initialized: now 1790632140968749 us; error 28 us; skew 500 ppm
Sep 28 21:49:00 dist-test-slave-jrck krb5kdc[3082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632140, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:00.975298 3457 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:49:00.975710 3457 webserver.cc:460] Webserver started at http://127.3.0.195:38269/ using document root <none> and password file <none>
I20260928 21:49:00.975939 3457 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:00.975989 3457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:00.976140 3457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:00.977223 3457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "5f83b7abba82466298fd174ee60e2549"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "34e1372398133c4e447647802d145551"
server_key_iv: "719450abde843c0041f5e44c3268a566"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.977548 3457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "5f83b7abba82466298fd174ee60e2549"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "34e1372398133c4e447647802d145551"
server_key_iv: "719450abde843c0041f5e44c3268a566"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.978745 3457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:49:00.979367 3472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:00.979530 3457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:00.979620 3457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "5f83b7abba82466298fd174ee60e2549"
format_stamp: "Formatted at 2026-09-28 21:49:00 on dist-test-slave-jrck"
server_key: "34e1372398133c4e447647802d145551"
server_key_iv: "719450abde843c0041f5e44c3268a566"
server_key_version: "encryptionkey@0"
I20260928 21:49:00.979679 3457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:00.994459 3457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:00.996541 3457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:00.996670 3457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:00.996894 3457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:00.997179 3457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:00.997210 3457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:00.997239 3457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:00.997263 3457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:01.003343 3457 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:34001
I20260928 21:49:01.003381 3585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:34001 every 8 connection(s)
I20260928 21:49:01.003691 3457 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:49:01.004629 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 3457
I20260928 21:49:01.004719 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:01.004873 3075 external_mini_cluster.cc:1428] Setting key 1ecb1d09b23916646e5c6daa073e7f7b
Sep 28 21:49:01 dist-test-slave-jrck krb5kdc[3082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632140, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:01.012198 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:01.004358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:41397 (local address 127.3.0.254:36557)
0928 21:49:01.004413 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:01.004415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:01.004611 (+ 196us) server_negotiation.cc:407] Connection header received
0928 21:49:01.005148 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:01.005149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:01.005159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:01.005237 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:01.006341 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:01.006860 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:01.007834 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:01.008050 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:01.008096 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:01.010102 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:01.010103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:01.010104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:01.010110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:01.011196 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:01.011515 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:01.011516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:01.011516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:01.011559 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:01.011725 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:01.011725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:01.011725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:01.011830 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:01.011874 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:01.012071 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:49:01.012132 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:49:01.012377 3588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:01.004344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36557 (local address 127.3.0.195:41397)
0928 21:49:01.004497 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:01.004538 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:01.005050 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:01.005313 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:01.005317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:01.006023 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:01.006215 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:01.006220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:01.006954 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:01.006955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:01.007762 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:01.007764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:01.008110 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:01.008111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:01.008210 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:01.008692 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:01.008700 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:01.010024 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:01.011256 (+ 1232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:01.011260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:01.011262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:01.011462 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:01.011602 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:01.011603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:01.011603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:01.011687 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:01.011872 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:01.011874 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:01.011981 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 21:49:01.012055 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:49:01.012169 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
I20260928 21:49:01.013248 3586 heartbeater.cc:347] Connected to a master server at 127.3.0.254:36557
I20260928 21:49:01.013345 3586 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:01.013521 3586 heartbeater.cc:511] Master 127.3.0.254:36557 requested a full tablet report, sending...
I20260928 21:49:01.013883 3127 ts_manager.cc:194] Registered new tserver with Master: 5f83b7abba82466298fd174ee60e2549 (127.3.0.195:34001)
I20260928 21:49:01.014226 3127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:41397
I20260928 21:49:01.016763 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:49:01 dist-test-slave-jrck krb5kdc[3082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:49:01 dist-test-slave-jrck krb5kdc[3082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:49:01.043092 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:01.033692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.3.0.193:39279)
0928 21:49:01.034169 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:01.034175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:01.034188 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:49:01.034217 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:01.034220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:01.034238 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:01.034317 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:01.034919 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:01.035997 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:01.037050 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:01.037594 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:01.037675 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:01.040203 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:01.040213 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:01.040213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:01.040221 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:01.041993 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:01.042302 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:01.042306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:01.042307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:01.042370 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:01.042528 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:01.042529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:01.042529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:01.042686 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:01.042857 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:01.042906 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 21:49:01.042985 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":73,"threads_started":1}
W20260928 21:49:01.043519 3232 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51306
Sep 28 21:49:01 dist-test-slave-jrck krb5kdc[3082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:49:01 dist-test-slave-jrck krb5kdc[3082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:49:01.068614 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:01.062001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.3.0.193:39279)
0928 21:49:01.062065 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:01.062067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:01.062111 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:49:01.062174 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:01.062175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:01.062186 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:01.062253 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:01.062781 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:01.063362 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:01.064040 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:01.064232 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:01.064276 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:01.066448 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:01.066450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:01.066450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:01.066457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:01.067688 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:01.068002 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:01.068003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:01.068004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:01.068053 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:01.068203 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:01.068204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:01.068204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:01.068293 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:01.068372 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:01.068440 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 21:49:01.068548 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:49:01.069094 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 3189
I20260928 21:49:01.073604 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 3323
I20260928 21:49:01.078336 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 3457
I20260928 21:49:01.081885 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 3095
2026-09-28T21:49:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1738 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:01 dist-test-slave-jrck krb5kdc[3600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:01 dist-test-slave-jrck krb5kdc[3600](info): set up 2 sockets
Sep 28 21:49:01 dist-test-slave-jrck krb5kdc[3600](info): commencing operation
krb5kdc: starting...
W20260928 21:49:02.011710 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.911s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:02.124577 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:43959
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:43959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:02.201741 3613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:02.201958 3613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:02.201989 3613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:02.203161 3613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:02.203203 3613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:02.203222 3613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:02.203238 3613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:02.203251 3613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:02.204757 3613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:43959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:43959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:02.205016 3613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:02.205226 3613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:02.209929 3619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:02.209940 3618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:02.209939 3621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:02.210090 3613 server_base.cc:1034] running on GCE node
I20260928 21:49:02.210461 3613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:02.210745 3613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:02.211881 3613 hybrid_clock.cc:648] HybridClock initialized: now 1790632142211868 us; error 29 us; skew 500 ppm
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:02.214210 3613 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:49:02.214660 3613 webserver.cc:460] Webserver started at http://127.3.0.254:40343/ using document root <none> and password file <none>
I20260928 21:49:02.214851 3613 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:02.214891 3613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:02.214983 3613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:02.215979 3613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "a068bb2bc36a43ae97d124e0d7da3c1f"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "02d9b77cd63ccc9b7033607400743311"
server_key_iv: "58a10bb18156c38facd8ba44f80fb1e2"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.216265 3613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "a068bb2bc36a43ae97d124e0d7da3c1f"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "02d9b77cd63ccc9b7033607400743311"
server_key_iv: "58a10bb18156c38facd8ba44f80fb1e2"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.217527 3613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:49:02.218207 3628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:02.218359 3613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:02.218438 3613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "a068bb2bc36a43ae97d124e0d7da3c1f"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "02d9b77cd63ccc9b7033607400743311"
server_key_iv: "58a10bb18156c38facd8ba44f80fb1e2"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.218504 3613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:02.235643 3613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:02.237739 3613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:02.237872 3613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:02.241706 3613 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:43959
I20260928 21:49:02.241748 3680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:43959 every 8 connection(s)
I20260928 21:49:02.242082 3613 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:02.242913 3681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:02.245260 3681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f: Bootstrap starting.
I20260928 21:49:02.245831 3681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:02.246098 3681 log.cc:824] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:02.246714 3681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f: No bootstrap required, opened a new log
I20260928 21:49:02.248274 3681 raft_consensus.cc:348] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a068bb2bc36a43ae97d124e0d7da3c1f" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43959 } }
I20260928 21:49:02.248421 3681 raft_consensus.cc:374] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:02.248479 3681 raft_consensus.cc:729] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068bb2bc36a43ae97d124e0d7da3c1f, State: Initialized, Role: FOLLOWER
I20260928 21:49:02.248636 3681 consensus_queue.cc:260] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a068bb2bc36a43ae97d124e0d7da3c1f" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43959 } }
I20260928 21:49:02.248696 3681 raft_consensus.cc:388] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:02.248716 3681 raft_consensus.cc:482] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:02.248724 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 3613
I20260928 21:49:02.248761 3681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:02.248807 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:02.248962 3075 external_mini_cluster.cc:1428] Setting key 28f39d56fc16e6b15a194a5e2a5e193b
I20260928 21:49:02.249369 3681 raft_consensus.cc:504] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a068bb2bc36a43ae97d124e0d7da3c1f" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43959 } }
I20260928 21:49:02.249477 3681 leader_election.cc:302] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a068bb2bc36a43ae97d124e0d7da3c1f; no voters:
I20260928 21:49:02.249616 3681 leader_election.cc:288] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:02.249867 3681 sys_catalog.cc:564] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:02.249902 3687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:02.250015 3687 raft_consensus.cc:686] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [term 1 LEADER]: Becoming Leader. State: Replica: a068bb2bc36a43ae97d124e0d7da3c1f, State: Running, Role: LEADER
I20260928 21:49:02.250104 3687 consensus_queue.cc:237] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a068bb2bc36a43ae97d124e0d7da3c1f" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43959 } }
I20260928 21:49:02.250458 3687 sys_catalog.cc:455] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a068bb2bc36a43ae97d124e0d7da3c1f. Latest consensus state: current_term: 1 leader_uuid: "a068bb2bc36a43ae97d124e0d7da3c1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a068bb2bc36a43ae97d124e0d7da3c1f" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43959 } } }
I20260928 21:49:02.250543 3687 sys_catalog.cc:458] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:02.250468 3688 sys_catalog.cc:455] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a068bb2bc36a43ae97d124e0d7da3c1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a068bb2bc36a43ae97d124e0d7da3c1f" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43959 } } }
I20260928 21:49:02.250638 3688 sys_catalog.cc:458] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:02.250734 3695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:02.251209 3695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:02.252729 3695 catalog_manager.cc:1347] Generated new cluster ID: 1bda3466f7614f2cb73844bf4692aa71
I20260928 21:49:02.252775 3695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:02.260104 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.249433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.3.0.254:43959)
0928 21:49:02.249625 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.249636 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.249672 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:49:02.249804 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.249811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.249867 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.250013 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:02.250900 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.252293 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.253316 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.253895 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.253993 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.256640 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:02.256646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:02.256646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:02.256659 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:02.258545 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.258899 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.258902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.258902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.259022 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.259194 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.259195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.259195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.259508 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:02.259658 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.259729 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.259882 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260928 21:49:02.275462 3695 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:02.276000 3695 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:02.280258 3695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a068bb2bc36a43ae97d124e0d7da3c1f: Generated new TSK 0
I20260928 21:49:02.280436 3695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:49:02.306514 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43959
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:02.384353 3707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:02.384586 3707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:02.384610 3707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:02.385841 3707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:02.385885 3707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:02.385915 3707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:02.387405 3707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43959
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:02.387630 3707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:02.387856 3707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:02.392553 3712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:02.392549 3713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:02.392611 3707 server_base.cc:1034] running on GCE node
W20260928 21:49:02.392549 3715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:02.392928 3707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:02.393151 3707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:02.394313 3707 hybrid_clock.cc:648] HybridClock initialized: now 1790632142394291 us; error 32 us; skew 500 ppm
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:02.396909 3707 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:49:02.397317 3707 webserver.cc:460] Webserver started at http://127.3.0.193:35041/ using document root <none> and password file <none>
I20260928 21:49:02.397500 3707 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:02.397538 3707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:02.397627 3707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:02.398593 3707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "720ab69b040e4731a5e1f54ab91df5db"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "a7fd91ca9c708da33f8ec96407dd7808"
server_key_iv: "f598ea3e0b1c598f1d4c64c9dc0f3007"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.398876 3707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "720ab69b040e4731a5e1f54ab91df5db"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "a7fd91ca9c708da33f8ec96407dd7808"
server_key_iv: "f598ea3e0b1c598f1d4c64c9dc0f3007"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.399963 3707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:02.400578 3722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:02.400713 3707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:02.400792 3707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "720ab69b040e4731a5e1f54ab91df5db"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "a7fd91ca9c708da33f8ec96407dd7808"
server_key_iv: "f598ea3e0b1c598f1d4c64c9dc0f3007"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.400861 3707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:02.411398 3707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:02.413120 3707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:02.413224 3707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:02.413385 3707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:02.413620 3707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:02.413650 3707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:02.413669 3707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:02.413681 3707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:02.418820 3707 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:40799
I20260928 21:49:02.418883 3835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:40799 every 8 connection(s)
I20260928 21:49:02.419154 3707 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:49:02.420486 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 3707
I20260928 21:49:02.420567 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:02.420710 3075 external_mini_cluster.cc:1428] Setting key 8dd7bbe0b65aa78915a4e34e2df75222
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
I20260928 21:49:02.429083 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.419769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:60515 (local address 127.3.0.254:43959)
0928 21:49:02.419809 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.419811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.419995 (+ 184us) server_negotiation.cc:407] Connection header received
0928 21:49:02.421065 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.421066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.421079 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.421125 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:02.421797 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.422525 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.423890 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.424081 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.424126 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.426497 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:02.426499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:02.426499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:02.426507 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:02.427760 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.428204 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.428204 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.428205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.428250 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.428532 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.428533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.428533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.428649 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:02.428697 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.428952 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.429017 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:49:02.429221 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.419747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:43959 (local address 127.3.0.193:60515)
0928 21:49:02.419881 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:02.419917 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:02.420983 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:02.421195 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:02.421199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:02.421516 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:02.421701 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.421706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.422612 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.422613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.423801 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.423803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.424145 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.424146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.424279 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.424835 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:02.424855 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:02.426399 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:02.427859 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:02.427865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:02.427868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:02.428137 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:02.428306 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:02.428307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:02.428307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:02.428443 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:02.428704 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:02.428709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:02.428826 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 21:49:02.428921 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:49:02.429041 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260928 21:49:02.429823 3836 heartbeater.cc:347] Connected to a master server at 127.3.0.254:43959
I20260928 21:49:02.429909 3836 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:02.430081 3836 heartbeater.cc:511] Master 127.3.0.254:43959 requested a full tablet report, sending...
I20260928 21:49:02.430635 3645 ts_manager.cc:194] Registered new tserver with Master: 720ab69b040e4731a5e1f54ab91df5db (127.3.0.193:40799)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
I20260928 21:49:02.431317 3645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:60515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:49:02.443111 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43959
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:02.522156 3841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:02.522358 3841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:02.522382 3841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:02.523514 3841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:02.523552 3841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:02.523581 3841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:02.525050 3841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43959
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:02.525260 3841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:02.525460 3841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:02.530259 3849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:02.530306 3841 server_base.cc:1034] running on GCE node
W20260928 21:49:02.530273 3846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:02.530269 3847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:02.530603 3841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:02.530853 3841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:02.531992 3841 hybrid_clock.cc:648] HybridClock initialized: now 1790632142531980 us; error 27 us; skew 500 ppm
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:02.534381 3841 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:49:02.534790 3841 webserver.cc:460] Webserver started at http://127.3.0.194:43611/ using document root <none> and password file <none>
I20260928 21:49:02.534964 3841 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:02.535003 3841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:02.535092 3841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:02.536074 3841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "1c8991d494014c7b95f8ed81db9189f1"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "0cb6590bb2b086f0abebd65f5ea537b0"
server_key_iv: "ded8097e5e3254b801b69b49d2d66e26"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.536367 3841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "1c8991d494014c7b95f8ed81db9189f1"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "0cb6590bb2b086f0abebd65f5ea537b0"
server_key_iv: "ded8097e5e3254b801b69b49d2d66e26"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.537606 3841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:02.538239 3856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:02.538380 3841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:02.538453 3841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "1c8991d494014c7b95f8ed81db9189f1"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "0cb6590bb2b086f0abebd65f5ea537b0"
server_key_iv: "ded8097e5e3254b801b69b49d2d66e26"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.538520 3841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:02.574738 3841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:02.576642 3841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:02.576771 3841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:02.577018 3841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:02.577283 3841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:02.577314 3841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:02.577335 3841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:02.577347 3841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:02.582512 3841 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:39773
I20260928 21:49:02.582580 3969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:39773 every 8 connection(s)
I20260928 21:49:02.582859 3841 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:49:02.587769 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 3841
I20260928 21:49:02.587857 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:02.588027 3075 external_mini_cluster.cc:1428] Setting key 269c7321989aacda81c1fc75748f1d9a
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:02.591617 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.583515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:53125 (local address 127.3.0.254:43959)
0928 21:49:02.583571 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.583573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.583761 (+ 188us) server_negotiation.cc:407] Connection header received
0928 21:49:02.584719 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.584721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.584730 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.584771 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:02.585345 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.585823 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.586724 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.586894 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.586935 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.588980 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:02.588981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:02.588981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:02.588987 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:02.590394 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.590746 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.590747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.590748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.590814 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.591030 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.591031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.591032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.591160 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:02.591222 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.591449 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.591530 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:49:02.591686 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.583481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:43959 (local address 127.3.0.194:53125)
0928 21:49:02.583640 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:02.583683 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:02.584617 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:02.584841 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:02.584845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:02.585092 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:02.585281 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.585285 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.585881 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.585882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.586667 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.586668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.586943 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.586943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.587038 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.587440 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:02.587448 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:02.588902 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:02.590468 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:02.590474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:02.590476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:02.590677 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:02.590890 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:02.590891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:02.590891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:02.590980 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:02.591213 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:02.591216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:02.591340 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 21:49:02.591417 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:49:02.591529 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260928 21:49:02.592178 3970 heartbeater.cc:347] Connected to a master server at 127.3.0.254:43959
I20260928 21:49:02.592268 3970 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:02.592438 3970 heartbeater.cc:511] Master 127.3.0.254:43959 requested a full tablet report, sending...
I20260928 21:49:02.592821 3645 ts_manager.cc:194] Registered new tserver with Master: 1c8991d494014c7b95f8ed81db9189f1 (127.3.0.194:39773)
I20260928 21:49:02.593173 3645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:53125
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:49:02.610355 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43959
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:46557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:02.687726 3975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:02.687930 3975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:02.687958 3975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:02.689091 3975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:02.689127 3975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:02.689152 3975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:02.690578 3975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:46557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43959
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:02.690768 3975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:02.690934 3975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:02.695680 3981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:02.695680 3983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:02.695725 3980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:02.695775 3975 server_base.cc:1034] running on GCE node
I20260928 21:49:02.696041 3975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:02.696277 3975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:02.697408 3975 hybrid_clock.cc:648] HybridClock initialized: now 1790632142697393 us; error 28 us; skew 500 ppm
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:02.699608 3975 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:49:02.700012 3975 webserver.cc:460] Webserver started at http://127.3.0.195:40793/ using document root <none> and password file <none>
I20260928 21:49:02.700222 3975 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:02.700268 3975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:02.700393 3975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:02.701382 3975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "42c54cd936bf4218b233bbe85833383d"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "77a911000aa5fa61e9143acb5c1577fb"
server_key_iv: "e7716761d54f2c599350559cb86ff1c0"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.701694 3975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "42c54cd936bf4218b233bbe85833383d"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "77a911000aa5fa61e9143acb5c1577fb"
server_key_iv: "e7716761d54f2c599350559cb86ff1c0"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.702862 3975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:02.703495 3990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:02.703639 3975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:02.703714 3975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "42c54cd936bf4218b233bbe85833383d"
format_stamp: "Formatted at 2026-09-28 21:49:02 on dist-test-slave-jrck"
server_key: "77a911000aa5fa61e9143acb5c1577fb"
server_key_iv: "e7716761d54f2c599350559cb86ff1c0"
server_key_version: "encryptionkey@0"
I20260928 21:49:02.703778 3975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:02.717221 3975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:02.718959 3975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:02.719094 3975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:02.719291 3975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:02.719591 3975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:02.719624 3975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:02.719653 3975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:02.719674 3975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:02.725387 3975 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:43653
I20260928 21:49:02.725423 4103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:43653 every 8 connection(s)
I20260928 21:49:02.725731 3975 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:02.733918 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.726388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:40645 (local address 127.3.0.254:43959)
0928 21:49:02.726440 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.726442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.726632 (+ 190us) server_negotiation.cc:407] Connection header received
0928 21:49:02.727548 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.727549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.727559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.727616 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:02.728183 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.728706 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.729647 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.729824 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.729866 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.731776 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:02.731778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:02.731778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:02.731784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:02.732874 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.733209 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.733210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.733210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.733253 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.733431 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.733431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.733431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.733535 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:02.733580 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.733799 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.733855 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:49:02.734088 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.726366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:43959 (local address 127.3.0.195:40645)
0928 21:49:02.726511 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:02.726553 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:02.727455 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:02.727675 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:02.727678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:02.727933 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:02.728122 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.728127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.728768 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.728769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.729576 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.729579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.729884 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.729885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.729986 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.730411 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:02.730420 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:02.731682 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:02.732940 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:02.732945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:02.732947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:02.733154 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:02.733300 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:02.733300 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:02.733301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:02.733390 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:02.733584 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:02.733588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:02.733702 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 21:49:02.733782 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:49:02.733900 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 21:49:02.734447 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 3975
I20260928 21:49:02.734544 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:02.734558 4104 heartbeater.cc:347] Connected to a master server at 127.3.0.254:43959
I20260928 21:49:02.734655 4104 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:02.734715 3075 external_mini_cluster.cc:1428] Setting key 5d833b2a208fd04bc33e10e1763f5dd1
I20260928 21:49:02.734846 4104 heartbeater.cc:511] Master 127.3.0.254:43959 requested a full tablet report, sending...
I20260928 21:49:02.735208 3645 ts_manager.cc:194] Registered new tserver with Master: 42c54cd936bf4218b233bbe85833383d (127.3.0.195:43653)
I20260928 21:49:02.735536 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:49:02.735673 3645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:40645
I20260928 21:49:02.742395 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.737070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.3.0.254:43959)
0928 21:49:02.737109 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.737111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.737241 (+ 130us) server_negotiation.cc:407] Connection header received
0928 21:49:02.737296 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.737297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.737307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.737356 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:02.737866 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.738323 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.739042 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.739227 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.739277 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.740255 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:02.740256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:02.740256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:02.740262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:02.741387 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.741680 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.741681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.741681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.741729 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.741889 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.741890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.741890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.742031 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:02.742117 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.742238 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.742304 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:49:02.744119 3645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:02.745640 3645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:02.751266 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.747436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:43653 (local address 127.0.0.1:53106)
0928 21:49:02.747685 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:02.747724 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:02.747812 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:02.748093 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:02.748094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:02.748096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:02.748146 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.748148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.749413 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.749413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.750154 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.750156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.750974 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.750975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.751051 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.751065 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 21:49:02.751131 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:49:02.751182 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":60,"threads_started":1}
I20260928 21:49:02.751344 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.747501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.3.0.195:43653)
0928 21:49:02.747730 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.747734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.747784 (+ 50us) server_negotiation.cc:407] Connection header received
0928 21:49:02.747926 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.747930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.747945 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.748019 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:02.748224 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.749291 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.750269 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.750907 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.750966 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.751021 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.751209 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.751245 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260928 21:49:02.751343 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.747532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.3.0.194:39773)
0928 21:49:02.747825 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.747829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.747837 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:02.747876 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.747878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.747894 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.747963 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:02.748079 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.748940 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.750362 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.750960 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.751023 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.751076 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.751223 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.751259 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":69,"threads_started":1}
I20260928 21:49:02.751538 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.747437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39773 (local address 127.0.0.1:45682)
0928 21:49:02.747730 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:02.747740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:02.747798 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:02.747963 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:02.747966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:02.747967 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:02.748026 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.748028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.749028 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.749029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.750168 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.750170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.751031 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.751032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.751092 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.751096 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:02.751142 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:49:02.751185 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":56,"threads_started":1}
I20260928 21:49:02.752029 4038 tablet_service.cc:1467] Processing CreateTablet for tablet 83ddb80e0e8e41229677c6b0c653c994 (DEFAULT_TABLE table=test-table [id=4864db973b80422994159bf79e9f9564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:02.752120 3904 tablet_service.cc:1467] Processing CreateTablet for tablet 83ddb80e0e8e41229677c6b0c653c994 (DEFAULT_TABLE table=test-table [id=4864db973b80422994159bf79e9f9564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:02.752363 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ddb80e0e8e41229677c6b0c653c994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:02.752414 3904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ddb80e0e8e41229677c6b0c653c994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:02.753021 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.747635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:40799 (local address 127.0.0.1:53800)
0928 21:49:02.747991 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:02.748000 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:02.748043 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:02.748220 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:02.748220 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:02.748222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:02.748255 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.748256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.749528 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.749529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.751832 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.751835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.752795 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.752796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.752855 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.752859 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:02.752913 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:49:02.752952 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":50,"threads_started":1}
I20260928 21:49:02.753278 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.747752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.3.0.193:40799)
0928 21:49:02.747949 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.747953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.748088 (+ 135us) server_negotiation.cc:407] Connection header received
0928 21:49:02.748127 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.748130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.748142 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.748214 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:02.748399 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.749438 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.751913 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.752643 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.752709 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.752761 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.752947 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.752991 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260928 21:49:02.754040 3770 tablet_service.cc:1467] Processing CreateTablet for tablet 83ddb80e0e8e41229677c6b0c653c994 (DEFAULT_TABLE table=test-table [id=4864db973b80422994159bf79e9f9564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:02.754329 3770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ddb80e0e8e41229677c6b0c653c994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:02.754914 4118 tablet_bootstrap.cc:492] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d: Bootstrap starting.
I20260928 21:49:02.755515 4118 tablet_bootstrap.cc:654] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:02.755540 4119 tablet_bootstrap.cc:492] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1: Bootstrap starting.
I20260928 21:49:02.756055 4118 log.cc:824] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:02.756157 4119 tablet_bootstrap.cc:654] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:02.756652 4119 log.cc:824] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:02.757050 4118 tablet_bootstrap.cc:492] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d: No bootstrap required, opened a new log
I20260928 21:49:02.757112 4118 ts_tablet_manager.cc:1397] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:02.757395 4119 tablet_bootstrap.cc:492] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1: No bootstrap required, opened a new log
I20260928 21:49:02.757377 4122 tablet_bootstrap.cc:492] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db: Bootstrap starting.
I20260928 21:49:02.757467 4119 ts_tablet_manager.cc:1397] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:02.758095 4122 tablet_bootstrap.cc:654] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:02.759177 4122 log.cc:824] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:02.759112 4118 raft_consensus.cc:348] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8991d494014c7b95f8ed81db9189f1" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39773 } } peers { permanent_uuid: "42c54cd936bf4218b233bbe85833383d" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43653 } } peers { permanent_uuid: "720ab69b040e4731a5e1f54ab91df5db" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 40799 } }
I20260928 21:49:02.759295 4118 raft_consensus.cc:374] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:02.759338 4118 raft_consensus.cc:729] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c54cd936bf4218b233bbe85833383d, State: Initialized, Role: FOLLOWER
I20260928 21:49:02.759297 4119 raft_consensus.cc:348] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8991d494014c7b95f8ed81db9189f1" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39773 } } peers { permanent_uuid: "42c54cd936bf4218b233bbe85833383d" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43653 } } peers { permanent_uuid: "720ab69b040e4731a5e1f54ab91df5db" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 40799 } }
I20260928 21:49:02.759464 4119 raft_consensus.cc:374] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:02.759506 4119 raft_consensus.cc:729] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c8991d494014c7b95f8ed81db9189f1, State: Initialized, Role: FOLLOWER
I20260928 21:49:02.759476 4118 consensus_queue.cc:260] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8991d494014c7b95f8ed81db9189f1" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39773 } } peers { permanent_uuid: "42c54cd936bf4218b233bbe85833383d" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43653 } } peers { permanent_uuid: "720ab69b040e4731a5e1f54ab91df5db" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 40799 } }
I20260928 21:49:02.759644 4119 consensus_queue.cc:260] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8991d494014c7b95f8ed81db9189f1" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39773 } } peers { permanent_uuid: "42c54cd936bf4218b233bbe85833383d" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43653 } } peers { permanent_uuid: "720ab69b040e4731a5e1f54ab91df5db" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 40799 } }
I20260928 21:49:02.759727 4104 heartbeater.cc:503] Master 127.3.0.254:43959 was elected leader, sending a full tablet report...
I20260928 21:49:02.759730 4118 ts_tablet_manager.cc:1428] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:49:02.759944 3970 heartbeater.cc:503] Master 127.3.0.254:43959 was elected leader, sending a full tablet report...
I20260928 21:49:02.760074 4119 ts_tablet_manager.cc:1428] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:49:02.760308 4122 tablet_bootstrap.cc:492] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db: No bootstrap required, opened a new log
I20260928 21:49:02.760380 4122 ts_tablet_manager.cc:1397] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:49:02.761911 4122 raft_consensus.cc:348] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8991d494014c7b95f8ed81db9189f1" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39773 } } peers { permanent_uuid: "42c54cd936bf4218b233bbe85833383d" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43653 } } peers { permanent_uuid: "720ab69b040e4731a5e1f54ab91df5db" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 40799 } }
I20260928 21:49:02.762085 4122 raft_consensus.cc:374] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:02.762131 4122 raft_consensus.cc:729] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720ab69b040e4731a5e1f54ab91df5db, State: Initialized, Role: FOLLOWER
I20260928 21:49:02.762245 4122 consensus_queue.cc:260] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8991d494014c7b95f8ed81db9189f1" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39773 } } peers { permanent_uuid: "42c54cd936bf4218b233bbe85833383d" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43653 } } peers { permanent_uuid: "720ab69b040e4731a5e1f54ab91df5db" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 40799 } }
I20260928 21:49:02.762477 4122 ts_tablet_manager.cc:1428] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:49:02.762494 3836 heartbeater.cc:503] Master 127.3.0.254:43959 was elected leader, sending a full tablet report...
I20260928 21:49:02.801954 4125 raft_consensus.cc:482] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:02.802074 4125 raft_consensus.cc:504] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8991d494014c7b95f8ed81db9189f1" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39773 } } peers { permanent_uuid: "42c54cd936bf4218b233bbe85833383d" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43653 } } peers { permanent_uuid: "720ab69b040e4731a5e1f54ab91df5db" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 40799 } }
I20260928 21:49:02.802448 4125 leader_election.cc:288] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42c54cd936bf4218b233bbe85833383d (127.3.0.195:43653), 720ab69b040e4731a5e1f54ab91df5db (127.3.0.193:40799)
I20260928 21:49:02.805542 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.802991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:60103 (local address 127.3.0.193:40799)
0928 21:49:02.803039 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.803040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.803086 (+ 46us) server_negotiation.cc:407] Connection header received
0928 21:49:02.803193 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.803194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.803208 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.803266 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:02.803420 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.803955 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.804879 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.805228 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.805285 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.805297 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.805435 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.805471 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:49:02.805546 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.802754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:37313 (local address 127.3.0.195:43653)
0928 21:49:02.802822 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.802824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.802832 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:02.802871 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.802872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.802884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.802931 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:02.803190 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.803770 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.804875 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.805236 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.805299 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.805309 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.805444 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.805482 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:49:02.805550 3972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.802650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:43653 (local address 127.3.0.194:37313)
0928 21:49:02.802717 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:02.802730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:02.802793 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:02.802957 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:02.802959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:02.802960 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:02.803073 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.803076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.803831 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.803833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.804781 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.804784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.805307 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.805308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.805381 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.805383 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:49:02.805430 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:49:02.805471 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 21:49:02.805557 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.802765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:40799 (local address 127.3.0.194:60103)
0928 21:49:02.803029 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:02.803039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:02.803109 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:02.803276 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:02.803277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:02.803279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:02.803364 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.803366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.804028 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.804029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.804800 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:02.804803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.805299 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.805299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:02.805372 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.805376 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:02.805421 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:49:02.805462 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
I20260928 21:49:02.805933 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ddb80e0e8e41229677c6b0c653c994" candidate_uuid: "1c8991d494014c7b95f8ed81db9189f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720ab69b040e4731a5e1f54ab91df5db" is_pre_election: true
I20260928 21:49:02.805938 4058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ddb80e0e8e41229677c6b0c653c994" candidate_uuid: "1c8991d494014c7b95f8ed81db9189f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c54cd936bf4218b233bbe85833383d" is_pre_election: true
I20260928 21:49:02.806108 4058 raft_consensus.cc:2454] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c8991d494014c7b95f8ed81db9189f1 in term 0.
I20260928 21:49:02.806111 3790 raft_consensus.cc:2454] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c8991d494014c7b95f8ed81db9189f1 in term 0.
I20260928 21:49:02.806316 3860 leader_election.cc:302] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8991d494014c7b95f8ed81db9189f1, 42c54cd936bf4218b233bbe85833383d; no voters:
I20260928 21:49:02.806473 4125 raft_consensus.cc:2781] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:02.806550 4125 raft_consensus.cc:482] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:02.806578 4125 raft_consensus.cc:3037] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:02.807235 4125 raft_consensus.cc:504] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8991d494014c7b95f8ed81db9189f1" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39773 } } peers { permanent_uuid: "42c54cd936bf4218b233bbe85833383d" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43653 } } peers { permanent_uuid: "720ab69b040e4731a5e1f54ab91df5db" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 40799 } }
I20260928 21:49:02.807350 4125 leader_election.cc:288] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [CANDIDATE]: Term 1 election: Requested vote from peers 42c54cd936bf4218b233bbe85833383d (127.3.0.195:43653), 720ab69b040e4731a5e1f54ab91df5db (127.3.0.193:40799)
I20260928 21:49:02.807503 4058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ddb80e0e8e41229677c6b0c653c994" candidate_uuid: "1c8991d494014c7b95f8ed81db9189f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42c54cd936bf4218b233bbe85833383d"
I20260928 21:49:02.807550 3790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ddb80e0e8e41229677c6b0c653c994" candidate_uuid: "1c8991d494014c7b95f8ed81db9189f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720ab69b040e4731a5e1f54ab91df5db"
I20260928 21:49:02.807582 4058 raft_consensus.cc:3037] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:02.807623 3790 raft_consensus.cc:3037] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:02.808442 4058 raft_consensus.cc:2454] T 83ddb80e0e8e41229677c6b0c653c994 P 42c54cd936bf4218b233bbe85833383d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c8991d494014c7b95f8ed81db9189f1 in term 1.
I20260928 21:49:02.808442 3790 raft_consensus.cc:2454] T 83ddb80e0e8e41229677c6b0c653c994 P 720ab69b040e4731a5e1f54ab91df5db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c8991d494014c7b95f8ed81db9189f1 in term 1.
I20260928 21:49:02.808610 3860 leader_election.cc:302] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c8991d494014c7b95f8ed81db9189f1, 720ab69b040e4731a5e1f54ab91df5db; no voters:
I20260928 21:49:02.808704 4125 raft_consensus.cc:2781] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:02.808871 4125 raft_consensus.cc:686] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [term 1 LEADER]: Becoming Leader. State: Replica: 1c8991d494014c7b95f8ed81db9189f1, State: Running, Role: LEADER
I20260928 21:49:02.808961 4125 consensus_queue.cc:237] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8991d494014c7b95f8ed81db9189f1" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39773 } } peers { permanent_uuid: "42c54cd936bf4218b233bbe85833383d" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43653 } } peers { permanent_uuid: "720ab69b040e4731a5e1f54ab91df5db" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 40799 } }
I20260928 21:49:02.809707 3645 catalog_manager.cc:5495] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c8991d494014c7b95f8ed81db9189f1 (127.3.0.194). New cstate: current_term: 1 leader_uuid: "1c8991d494014c7b95f8ed81db9189f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c8991d494014c7b95f8ed81db9189f1" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42c54cd936bf4218b233bbe85833383d" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "720ab69b040e4731a5e1f54ab91df5db" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 40799 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:49:02.830488 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.823735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.3.0.193:40799)
0928 21:49:02.823777 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.823779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.823881 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:49:02.823949 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.823950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.823959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.824000 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:02.824540 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.825018 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.825711 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.825909 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.825963 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.828046 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:02.828059 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:02.828060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:02.828067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:02.829592 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.829883 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.829886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.829887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.829939 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.830063 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.830063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.830064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.830191 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:02.830296 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.830351 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.830411 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 21:49:02.833366 3971 tablet.cc:2369] T 83ddb80e0e8e41229677c6b0c653c994 P 1c8991d494014c7b95f8ed81db9189f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:49:02.851883 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.845340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.3.0.193:40799)
0928 21:49:02.845413 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.845415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.845430 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:49:02.845512 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.845513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.845523 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.845576 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:02.846123 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.846628 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.847332 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.847519 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.847574 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.849681 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:02.849682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:02.849683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:02.849690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:02.850841 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.851145 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.851146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.851147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.851217 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.851370 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.851371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.851372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.851465 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:02.851549 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.851717 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.851792 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[3600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:49:02.872035 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:02.866164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.3.0.193:40799)
0928 21:49:02.866234 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:02.866236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:02.866289 (+ 53us) server_negotiation.cc:407] Connection header received
0928 21:49:02.866355 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:02.866356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:02.866365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:02.866411 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:02.866878 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.867329 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.868000 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:02.868171 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:02.868211 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:02.870222 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:02.870224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:02.870224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:02.870230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:02.871261 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.871526 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.871526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.871527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.871569 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:02.871686 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:02.871686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:02.871687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:02.871765 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:02.871828 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:02.871914 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 21:49:02.871965 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 21:49:02.872538 3750 tablet_service.cc:2908] Rejecting scan request for tablet 83ddb80e0e8e41229677c6b0c653c994: Uninitialized: clean time has not yet been initialized
I20260928 21:49:02.873243 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 3707
I20260928 21:49:02.877426 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 3841
I20260928 21:49:02.881424 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 3975
I20260928 21:49:02.885059 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 3613
2026-09-28T21:49:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1804 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[4142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[4142](info): set up 2 sockets
Sep 28 21:49:02 dist-test-slave-jrck krb5kdc[4142](info): commencing operation
krb5kdc: starting...
W20260928 21:49:03.831776 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:03 dist-test-slave-jrck krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:03.942864 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33975
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:33975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:04.021669 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:04.021890 4155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:04.021924 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:04.023087 4155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:04.023128 4155 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:49:04.023144 4155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:04.023161 4155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:04.023177 4155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:04.023200 4155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:04.024716 4155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:33975
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:04.024961 4155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:04.025161 4155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:04.029712 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:04.029721 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:04.029935 4155 server_base.cc:1034] running on GCE node
W20260928 21:49:04.029713 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:04.030237 4155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:04.030504 4155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:04.031672 4155 hybrid_clock.cc:648] HybridClock initialized: now 1790632144031649 us; error 31 us; skew 500 ppm
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:04.034186 4155 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:49:04.034622 4155 webserver.cc:460] Webserver started at http://127.3.0.254:35423/ using document root <none> and password file <none>
I20260928 21:49:04.034839 4155 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:04.034888 4155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:04.035013 4155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:04.036075 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "d96b4530e23943d19996ae3bcf60ecc7"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "cc0ffac6102bcdbf10f77292dfe185d7"
server_key_iv: "85024d4162b985e8eeac69b08468a0c5"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.036418 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "d96b4530e23943d19996ae3bcf60ecc7"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "cc0ffac6102bcdbf10f77292dfe185d7"
server_key_iv: "85024d4162b985e8eeac69b08468a0c5"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.037823 4155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:04.038707 4170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.038908 4155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:04.039001 4155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "d96b4530e23943d19996ae3bcf60ecc7"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "cc0ffac6102bcdbf10f77292dfe185d7"
server_key_iv: "85024d4162b985e8eeac69b08468a0c5"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.039069 4155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:04.074038 4155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:04.075852 4155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:04.075996 4155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:04.080266 4155 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:33975
I20260928 21:49:04.080322 4232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:33975 every 8 connection(s)
I20260928 21:49:04.080667 4155 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:04.081302 4233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:04.083529 4233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7: Bootstrap starting.
I20260928 21:49:04.084092 4233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:04.084357 4233 log.cc:824] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:04.085018 4233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7: No bootstrap required, opened a new log
I20260928 21:49:04.087033 4233 raft_consensus.cc:348] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96b4530e23943d19996ae3bcf60ecc7" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33975 } }
I20260928 21:49:04.087173 4233 raft_consensus.cc:374] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:04.087209 4233 raft_consensus.cc:729] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96b4530e23943d19996ae3bcf60ecc7, State: Initialized, Role: FOLLOWER
I20260928 21:49:04.087354 4233 consensus_queue.cc:260] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96b4530e23943d19996ae3bcf60ecc7" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33975 } }
I20260928 21:49:04.087414 4233 raft_consensus.cc:388] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:04.087405 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 4155
I20260928 21:49:04.087437 4233 raft_consensus.cc:482] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:04.087474 4233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:04.087495 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:04.087625 3075 external_mini_cluster.cc:1428] Setting key e625d0ec3a01e7953add58b8f5cbaffd
I20260928 21:49:04.088080 4233 raft_consensus.cc:504] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96b4530e23943d19996ae3bcf60ecc7" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33975 } }
I20260928 21:49:04.088186 4233 leader_election.cc:302] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d96b4530e23943d19996ae3bcf60ecc7; no voters:
I20260928 21:49:04.088337 4233 leader_election.cc:288] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:04.088368 4238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:04.088517 4238 raft_consensus.cc:686] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [term 1 LEADER]: Becoming Leader. State: Replica: d96b4530e23943d19996ae3bcf60ecc7, State: Running, Role: LEADER
I20260928 21:49:04.088591 4233 sys_catalog.cc:564] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:04.088605 4238 consensus_queue.cc:237] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96b4530e23943d19996ae3bcf60ecc7" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33975 } }
I20260928 21:49:04.089541 4240 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d96b4530e23943d19996ae3bcf60ecc7. Latest consensus state: current_term: 1 leader_uuid: "d96b4530e23943d19996ae3bcf60ecc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96b4530e23943d19996ae3bcf60ecc7" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33975 } } }
I20260928 21:49:04.089627 4240 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [sys.catalog]: This master's current role is: LEADER
W20260928 21:49:04.090253 4253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:49:04.090301 4253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:49:04.090013 4239 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d96b4530e23943d19996ae3bcf60ecc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96b4530e23943d19996ae3bcf60ecc7" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33975 } } }
I20260928 21:49:04.090406 4239 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:04.090349 4252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:04.090598 4252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:04.092001 4252 catalog_manager.cc:1347] Generated new cluster ID: 89a6338312684bf9ad6cbe4cf26adf47
I20260928 21:49:04.092047 4252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:04.098299 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.088072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.3.0.254:33975)
0928 21:49:04.088266 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.088276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.088321 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:49:04.088443 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.088473 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.088532 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.088675 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.089310 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.090712 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.091572 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.092170 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.092273 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.094891 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.094895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.094895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.094907 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.096758 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.097139 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.097141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.097141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.097239 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.097437 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.097437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.097437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.097721 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.097873 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.097962 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.098092 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260928 21:49:04.102046 4252 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:04.102654 4252 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:04.111805 4252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7: Generated new TSK 0
I20260928 21:49:04.112005 4252 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:49:04.135264 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33975
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:04.213685 4259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:04.213876 4259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:04.213896 4259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:04.214987 4259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:04.215021 4259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:04.215040 4259 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:49:04.215059 4259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:04.216414 4259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.0.254:33975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:04.216629 4259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:04.216820 4259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:04.221385 4267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:04.221455 4265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:04.221606 4259 server_base.cc:1034] running on GCE node
W20260928 21:49:04.221402 4264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:04.221879 4259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:04.222118 4259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:04.223253 4259 hybrid_clock.cc:648] HybridClock initialized: now 1790632144223239 us; error 29 us; skew 500 ppm
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:04.225579 4259 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:49:04.226006 4259 webserver.cc:460] Webserver started at http://127.3.0.193:34085/ using document root <none> and password file <none>
I20260928 21:49:04.226230 4259 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:04.226277 4259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:04.226405 4259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:04.227420 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "900f4a9cf36e4af5aba0e56e35f910ab"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "a2e6e139d004a75d6f990806d3cc885a"
server_key_iv: "68645a7dbf74e88c46b78b2e01d9303e"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.227732 4259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "900f4a9cf36e4af5aba0e56e35f910ab"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "a2e6e139d004a75d6f990806d3cc885a"
server_key_iv: "68645a7dbf74e88c46b78b2e01d9303e"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.229089 4259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:04.229789 4274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.229966 4259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:04.230046 4259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "900f4a9cf36e4af5aba0e56e35f910ab"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "a2e6e139d004a75d6f990806d3cc885a"
server_key_iv: "68645a7dbf74e88c46b78b2e01d9303e"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.230110 4259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:04.240162 4259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:04.242157 4259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:04.242291 4259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:04.242851 4259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:04.242900 4259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.242928 4259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:04.242951 4259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:04.249465 4259 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:45151
I20260928 21:49:04.249532 4389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:45151 every 8 connection(s)
I20260928 21:49:04.249835 4259 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:49:04.251746 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.243256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:34201 (local address 127.3.0.254:33975)
0928 21:49:04.243305 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.243308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.243532 (+ 224us) server_negotiation.cc:407] Connection header received
0928 21:49:04.244037 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.244039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.244052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.244123 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.244795 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.245556 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.246540 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.246764 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.246824 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.249257 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.249258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.249259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.249269 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.250721 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.251068 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.251069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.251069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.251113 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.251283 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.251283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.251284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.251379 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.251423 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.251626 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.251684 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:49:04.251915 4283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.243246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33975 (local address 127.3.0.193:34201)
0928 21:49:04.243407 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.243447 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.243949 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.244189 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.244194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.244518 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:04.244719 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.244724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.245646 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.245647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.246446 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.246448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.246834 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.246835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.246973 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.247421 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:04.247429 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:04.249155 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:04.250797 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.250801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.250803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.251011 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.251156 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.251157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.251157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.251242 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.251422 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:04.251425 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:04.251532 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.251609 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.251721 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260928 21:49:04.252332 4390 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33975
I20260928 21:49:04.252403 4390 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:04.252578 4390 heartbeater.cc:511] Master 127.3.0.254:33975 requested a full tablet report, sending...
I20260928 21:49:04.253087 4186 ts_manager.cc:194] Registered new tserver with Master: 900f4a9cf36e4af5aba0e56e35f910ab (127.3.0.193:45151)
I20260928 21:49:04.253804 4186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:34201
I20260928 21:49:04.258035 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.253393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:58399 (local address 127.3.0.254:33975)
0928 21:49:04.253433 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.253435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.253491 (+ 56us) server_negotiation.cc:407] Connection header received
0928 21:49:04.253535 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.253536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.253544 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.253603 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.253849 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.254407 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.255090 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.255265 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.255313 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.256008 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.256009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.256009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.256014 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.257215 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.257491 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.257492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.257492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.257546 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.257694 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.257694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.257695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.257784 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.257828 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.257922 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.257980 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:04.258036 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.253323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33975 (local address 127.3.0.193:58399)
0928 21:49:04.253440 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.253450 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.253500 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.253607 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.253607 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.253746 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:04.253791 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.253792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.254496 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.254497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.255024 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.255026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.255313 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.255313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.255363 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.255672 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:04.255676 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:04.255939 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:04.257276 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.257277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.257277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.257417 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.257605 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.257606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.257606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.257647 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.257823 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:04.257823 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:04.257853 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.257900 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.257968 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260928 21:49:04.259541 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 4259
I20260928 21:49:04.259615 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:04.259742 3075 external_mini_cluster.cc:1428] Setting key 88cccb13fa2e8d7745b3222cf9e6a270
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:49:04.282796 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33975
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:04.360100 4399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:04.360309 4399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:04.360339 4399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:04.361526 4399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:04.361568 4399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:04.361595 4399 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:49:04.361619 4399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:04.363023 4399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.0.254:33975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:04.363242 4399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:04.363451 4399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:04.368119 4407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:04.368130 4405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:04.368126 4404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:04.368268 4399 server_base.cc:1034] running on GCE node
I20260928 21:49:04.368443 4399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:04.368690 4399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:04.369818 4399 hybrid_clock.cc:648] HybridClock initialized: now 1790632144369802 us; error 31 us; skew 500 ppm
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:04.372093 4399 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:49:04.372509 4399 webserver.cc:460] Webserver started at http://127.3.0.194:43033/ using document root <none> and password file <none>
I20260928 21:49:04.372717 4399 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:04.372763 4399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:04.372892 4399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:04.373873 4399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "4d15fc17020842d294fd650ba9c2d532"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "a052bb387f22c73812376df3433fa8d3"
server_key_iv: "e8232126912135f8d6eb903b5c52f794"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.374191 4399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "4d15fc17020842d294fd650ba9c2d532"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "a052bb387f22c73812376df3433fa8d3"
server_key_iv: "e8232126912135f8d6eb903b5c52f794"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.375430 4399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:49:04.376075 4414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.376240 4399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:04.376309 4399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "4d15fc17020842d294fd650ba9c2d532"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "a052bb387f22c73812376df3433fa8d3"
server_key_iv: "e8232126912135f8d6eb903b5c52f794"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.376375 4399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:04.394800 4399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:04.400019 4399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:04.400161 4399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:04.400785 4399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:04.400839 4399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.400877 4399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:04.400893 4399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:04.407505 4399 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:45937
I20260928 21:49:04.407639 4529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:45937 every 8 connection(s)
I20260928 21:49:04.407912 4399 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:49:04.409540 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.401290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:39445 (local address 127.3.0.254:33975)
0928 21:49:04.401337 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.401339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.401523 (+ 184us) server_negotiation.cc:407] Connection header received
0928 21:49:04.402018 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.402020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.402032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.402090 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.402768 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.403277 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.404322 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.404611 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.404677 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.407144 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.407146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.407147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.407154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.408567 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.408895 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.408895 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.408896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.408943 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.409092 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.409093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.409093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.409181 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.409219 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.409419 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.409477 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:04.409699 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.401222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33975 (local address 127.3.0.194:39445)
0928 21:49:04.401392 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.401441 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.401927 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.402184 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.402187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.402479 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:04.402681 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.402686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.403358 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.403359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.404240 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.404242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.404679 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.404680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.404826 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.405331 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:04.405342 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:04.407028 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:04.408623 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.408626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.408628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.408838 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.408979 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.408979 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.408980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.409059 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.409215 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:04.409217 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:04.409323 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.409394 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.409502 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260928 21:49:04.410108 4530 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33975
I20260928 21:49:04.410187 4530 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:04.410344 4530 heartbeater.cc:511] Master 127.3.0.254:33975 requested a full tablet report, sending...
I20260928 21:49:04.410710 4187 ts_manager.cc:194] Registered new tserver with Master: 4d15fc17020842d294fd650ba9c2d532 (127.3.0.194:45937)
I20260928 21:49:04.411175 4187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:39445
I20260928 21:49:04.415823 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.411067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:51185 (local address 127.3.0.254:33975)
0928 21:49:04.411103 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.411105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.411370 (+ 265us) server_negotiation.cc:407] Connection header received
0928 21:49:04.411435 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.411436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.411445 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.411489 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.411762 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.412218 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.412920 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.413116 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.413169 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.413883 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.413884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.413884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.413891 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.414829 (+ 938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.415125 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.415126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.415126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.415178 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.415333 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.415334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.415335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.415472 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.415529 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.415660 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.415728 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:49:04.415838 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.411021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33975 (local address 127.3.0.194:51185)
0928 21:49:04.411298 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.411307 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.411379 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.411492 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.411493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.411652 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:04.411698 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.411699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.412299 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.412300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.412851 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.412852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.413172 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.413173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.413230 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.413552 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:04.413555 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:04.413800 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:04.414917 (+ 1117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.414918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.414918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.415068 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.415228 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.415229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.415229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.415284 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.415529 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:04.415530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:04.415573 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.415632 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.415729 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":71,"threads_started":1}
I20260928 21:49:04.417284 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 4399
I20260928 21:49:04.417351 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:04.417491 3075 external_mini_cluster.cc:1428] Setting key 8a7891125508ed12381d47d9691582f9
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:49:04.439082 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33975
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:04.517143 4539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:04.517359 4539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:04.517391 4539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:04.518561 4539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:04.518600 4539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:04.518628 4539 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:49:04.518651 4539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:04.520047 4539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.0.254:33975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:04.520298 4539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:04.520526 4539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:04.525148 4544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:04.525113 4547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:04.525106 4545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:04.525503 4539 server_base.cc:1034] running on GCE node
I20260928 21:49:04.525695 4539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:04.525900 4539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:04.527032 4539 hybrid_clock.cc:648] HybridClock initialized: now 1790632144527018 us; error 27 us; skew 500 ppm
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:04.529384 4539 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:49:04.529791 4539 webserver.cc:460] Webserver started at http://127.3.0.195:38525/ using document root <none> and password file <none>
I20260928 21:49:04.529991 4539 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:04.530037 4539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:04.530155 4539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:04.531193 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "803f885391e24048af4bb96e664d37da"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "5acf665192d58fe3b565f9303d0eede5"
server_key_iv: "daf6a318dbd410bb068a2e2750cbbfd7"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.531522 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "803f885391e24048af4bb96e664d37da"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "5acf665192d58fe3b565f9303d0eede5"
server_key_iv: "daf6a318dbd410bb068a2e2750cbbfd7"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.532729 4539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:49:04.533440 4554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.533588 4539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:04.533669 4539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "803f885391e24048af4bb96e664d37da"
format_stamp: "Formatted at 2026-09-28 21:49:04 on dist-test-slave-jrck"
server_key: "5acf665192d58fe3b565f9303d0eede5"
server_key_iv: "daf6a318dbd410bb068a2e2750cbbfd7"
server_key_version: "encryptionkey@0"
I20260928 21:49:04.533727 4539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:04.543215 4539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:04.544991 4539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:04.545118 4539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:04.545588 4539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:04.545626 4539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.545653 4539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:04.545671 4539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.551787 4539 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:33325
I20260928 21:49:04.551906 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:33325 every 8 connection(s)
I20260928 21:49:04.552156 4539 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:04.553248 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 4539
I20260928 21:49:04.553361 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:04.553537 3075 external_mini_cluster.cc:1428] Setting key 70e54c7bb8ffa5c99f4fd31a1724c7cf
I20260928 21:49:04.555162 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.545991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:34869 (local address 127.3.0.254:33975)
0928 21:49:04.546052 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.546054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.546244 (+ 190us) server_negotiation.cc:407] Connection header received
0928 21:49:04.546719 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.546720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.546733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.546811 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.547572 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.548299 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.549326 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.549624 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.549693 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.552740 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.552741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.552741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.552748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.554042 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.554480 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.554481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.554481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.554531 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.554709 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.554710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.554710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.554804 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.554849 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.555040 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.555100 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:49:04.555351 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.545981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33975 (local address 127.3.0.195:34869)
0928 21:49:04.546119 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.546159 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.546630 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.546871 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.546875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.547181 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:04.547475 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.547482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.548392 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.548393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.549237 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.549240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.549721 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.549722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.549880 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.550862 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:04.550873 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:04.552636 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:04.554107 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.554114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.554116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.554422 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.554578 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.554579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.554579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.554669 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.554845 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:04.554848 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:04.554953 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.555027 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.555140 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":87,"threads_started":1}
I20260928 21:49:04.555711 4670 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33975
I20260928 21:49:04.555783 4670 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:04.555938 4670 heartbeater.cc:511] Master 127.3.0.254:33975 requested a full tablet report, sending...
I20260928 21:49:04.556252 4187 ts_manager.cc:194] Registered new tserver with Master: 803f885391e24048af4bb96e664d37da (127.3.0.195:33325)
I20260928 21:49:04.556701 4187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:34869
I20260928 21:49:04.561285 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.556593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33975 (local address 127.3.0.195:46879)
0928 21:49:04.556745 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.556755 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.556827 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.556939 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.556939 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.557123 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:04.557174 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.557175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.557860 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.557860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.558351 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.558352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.558624 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.558625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.558672 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.559014 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:04.559018 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:04.559262 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:04.560444 (+ 1182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.560445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.560446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.560625 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.560842 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.560842 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.560843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.560888 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.561067 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:04.561068 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:04.561099 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.561141 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.561215 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260928 21:49:04.561302 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.556645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:46879 (local address 127.3.0.254:33975)
0928 21:49:04.556690 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.556691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.556805 (+ 114us) server_negotiation.cc:407] Connection header received
0928 21:49:04.556877 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.556878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.556887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.556934 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.557249 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.557781 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.558410 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.558579 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.558620 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.559327 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.559328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.559328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.559334 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.560386 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.560728 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.560729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.560729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.560786 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.560931 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.560931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.560932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.561021 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.561074 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.561157 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.561217 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:49:04.565135 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:04.595498 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.588098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.3.0.254:33975)
0928 21:49:04.588185 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.588188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.588199 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:49:04.588232 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.588233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.588246 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.588328 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.589163 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.589860 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.590884 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.591103 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.591160 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.593387 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.593389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.593389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.593396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.594615 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.594936 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.594937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.594937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.594983 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.595133 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.595133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.595133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.595222 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.595287 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.595373 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.595427 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:49:04.596971 4187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:04.598474 4187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:04.604110 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.600480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:33325 (local address 127.0.0.1:42546)
0928 21:49:04.600789 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.600796 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.600831 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.601194 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.601195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.601197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:04.601246 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.601247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.602046 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.602047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.602997 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.603000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.603842 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.603843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.603923 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.603939 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.603985 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.604033 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":30,"thread_start_us":57,"threads_started":1}
I20260928 21:49:04.604535 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.600896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.3.0.195:33325)
0928 21:49:04.601042 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.601047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.601062 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:49:04.601100 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.601102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.601117 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.601187 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:04.601299 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.601972 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.603082 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.603748 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.604300 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.604345 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.604375 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.604404 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260928 21:49:04.604794 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.600475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:45151 (local address 127.0.0.1:53698)
0928 21:49:04.600588 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.600631 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.600694 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.601056 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.601059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.601061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:04.601131 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.601133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.602130 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.602131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.603318 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.603321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.604443 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.604444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.604552 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.604557 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.604685 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.604724 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260928 21:49:04.604948 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.600721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.3.0.193:45151)
0928 21:49:04.600876 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.600881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.600892 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:49:04.600932 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.600935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.600951 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.601207 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:04.601270 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.602049 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.603403 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.604117 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.604185 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.604250 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.604798 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.604844 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260928 21:49:04.605183 4604 tablet_service.cc:1467] Processing CreateTablet for tablet 05ffe6cfa7e843df928ad57e90ae4b36 (DEFAULT_TABLE table=test-table [id=ab7882fb522e47aab4e7b90bec33b989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:04.605575 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ffe6cfa7e843df928ad57e90ae4b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:04.605639 4313 tablet_service.cc:1467] Processing CreateTablet for tablet 05ffe6cfa7e843df928ad57e90ae4b36 (DEFAULT_TABLE table=test-table [id=ab7882fb522e47aab4e7b90bec33b989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:04.605952 4313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ffe6cfa7e843df928ad57e90ae4b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:04.608642 4690 tablet_bootstrap.cc:492] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: Bootstrap starting.
I20260928 21:49:04.609040 4691 tablet_bootstrap.cc:492] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: Bootstrap starting.
I20260928 21:49:04.609117 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.600489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:45937 (local address 127.0.0.1:34160)
0928 21:49:04.600893 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.600903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.600952 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.602331 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.602332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.602334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:04.602369 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.602370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.603512 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.603513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.604199 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.604201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.605256 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.605257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.608945 (+ 3688us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.608952 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.609006 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.609049 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":51,"threads_started":1}
I20260928 21:49:04.609421 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.600718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.3.0.194:45937)
0928 21:49:04.602182 (+ 1464us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.602187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.602196 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:49:04.602230 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.602231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.602246 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.602328 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:04.602433 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.603453 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.604292 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.605115 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.605170 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.605225 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.609273 (+ 4048us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.609323 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1429,"thread_start_us":80,"threads_started":1}
I20260928 21:49:04.609508 4690 tablet_bootstrap.cc:654] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:04.609656 4691 tablet_bootstrap.cc:654] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:04.609891 4690 log.cc:824] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:04.610013 4464 tablet_service.cc:1467] Processing CreateTablet for tablet 05ffe6cfa7e843df928ad57e90ae4b36 (DEFAULT_TABLE table=test-table [id=ab7882fb522e47aab4e7b90bec33b989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:04.610113 4691 log.cc:824] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:04.610320 4464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ffe6cfa7e843df928ad57e90ae4b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:04.610585 4690 tablet_bootstrap.cc:492] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: No bootstrap required, opened a new log
I20260928 21:49:04.610644 4690 ts_tablet_manager.cc:1397] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:49:04.612287 4690 raft_consensus.cc:348] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } }
I20260928 21:49:04.612432 4690 raft_consensus.cc:374] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:04.612560 4690 raft_consensus.cc:729] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803f885391e24048af4bb96e664d37da, State: Initialized, Role: FOLLOWER
I20260928 21:49:04.612682 4690 consensus_queue.cc:260] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } }
I20260928 21:49:04.612934 4690 ts_tablet_manager.cc:1428] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:49:04.612939 4670 heartbeater.cc:503] Master 127.3.0.254:33975 was elected leader, sending a full tablet report...
I20260928 21:49:04.613072 4691 tablet_bootstrap.cc:492] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: No bootstrap required, opened a new log
I20260928 21:49:04.613139 4691 ts_tablet_manager.cc:1397] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 21:49:04.613399 4695 tablet_bootstrap.cc:492] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: Bootstrap starting.
I20260928 21:49:04.614024 4695 tablet_bootstrap.cc:654] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:04.614308 4695 log.cc:824] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:04.614737 4691 raft_consensus.cc:348] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } }
I20260928 21:49:04.614861 4691 raft_consensus.cc:374] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:04.614882 4691 raft_consensus.cc:729] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900f4a9cf36e4af5aba0e56e35f910ab, State: Initialized, Role: FOLLOWER
I20260928 21:49:04.614890 4695 tablet_bootstrap.cc:492] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: No bootstrap required, opened a new log
I20260928 21:49:04.614953 4695 ts_tablet_manager.cc:1397] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:04.614974 4691 consensus_queue.cc:260] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } }
I20260928 21:49:04.615203 4691 ts_tablet_manager.cc:1428] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:49:04.615267 4390 heartbeater.cc:503] Master 127.3.0.254:33975 was elected leader, sending a full tablet report...
I20260928 21:49:04.616674 4695 raft_consensus.cc:348] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } }
I20260928 21:49:04.616794 4695 raft_consensus.cc:374] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:04.616823 4695 raft_consensus.cc:729] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d15fc17020842d294fd650ba9c2d532, State: Initialized, Role: FOLLOWER
I20260928 21:49:04.616928 4695 consensus_queue.cc:260] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } }
I20260928 21:49:04.617136 4695 ts_tablet_manager.cc:1428] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:49:04.617187 4530 heartbeater.cc:503] Master 127.3.0.254:33975 was elected leader, sending a full tablet report...
I20260928 21:49:04.625974 4694 raft_consensus.cc:482] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:04.626075 4694 raft_consensus.cc:504] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } }
I20260928 21:49:04.626346 4694 leader_election.cc:288] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900f4a9cf36e4af5aba0e56e35f910ab (127.3.0.193:45151), 4d15fc17020842d294fd650ba9c2d532 (127.3.0.194:45937)
I20260928 21:49:04.629513 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.626596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:52507 (local address 127.3.0.193:45151)
0928 21:49:04.626639 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.626642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.626804 (+ 162us) server_negotiation.cc:407] Connection header received
0928 21:49:04.626851 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.626853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.626865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.626914 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:04.627023 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.627517 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.628820 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.629170 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.629225 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.629234 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.629397 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.629443 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:49:04.629526 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.626513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:45937 (local address 127.3.0.195:55107)
0928 21:49:04.626555 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.626565 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.626604 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.626752 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.626753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.626755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:04.626823 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.626824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.627525 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.627526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.628743 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.628745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.629286 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.629287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.629362 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.629365 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.629416 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.629453 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":1}
I20260928 21:49:04.629521 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.626601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:55107 (local address 127.3.0.194:45937)
0928 21:49:04.626639 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.626642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.626649 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:49:04.626674 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.626675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.626684 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.626746 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:04.626910 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.627454 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.628871 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.629212 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.629279 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.629289 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.629428 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.629464 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:49:04.629527 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.626512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:45151 (local address 127.3.0.195:52507)
0928 21:49:04.626744 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.626754 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.626807 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.626915 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.626916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.626917 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:04.626975 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.626976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.627592 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.627592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.628743 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.628745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.629247 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.629248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.629323 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.629327 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.629377 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.629411 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":82,"thread_start_us":105,"threads_started":1}
I20260928 21:49:04.629853 4484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ffe6cfa7e843df928ad57e90ae4b36" candidate_uuid: "803f885391e24048af4bb96e664d37da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d15fc17020842d294fd650ba9c2d532" is_pre_election: true
I20260928 21:49:04.629884 4344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ffe6cfa7e843df928ad57e90ae4b36" candidate_uuid: "803f885391e24048af4bb96e664d37da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" is_pre_election: true
I20260928 21:49:04.630003 4484 raft_consensus.cc:2454] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 803f885391e24048af4bb96e664d37da in term 0.
I20260928 21:49:04.630003 4344 raft_consensus.cc:2454] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 803f885391e24048af4bb96e664d37da in term 0.
I20260928 21:49:04.630205 4558 leader_election.cc:302] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 803f885391e24048af4bb96e664d37da, 900f4a9cf36e4af5aba0e56e35f910ab; no voters:
I20260928 21:49:04.630374 4694 raft_consensus.cc:2781] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:04.630430 4694 raft_consensus.cc:482] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:04.630445 4694 raft_consensus.cc:3037] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:04.631070 4694 raft_consensus.cc:504] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } }
I20260928 21:49:04.631177 4694 leader_election.cc:288] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [CANDIDATE]: Term 1 election: Requested vote from peers 900f4a9cf36e4af5aba0e56e35f910ab (127.3.0.193:45151), 4d15fc17020842d294fd650ba9c2d532 (127.3.0.194:45937)
I20260928 21:49:04.631330 4484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ffe6cfa7e843df928ad57e90ae4b36" candidate_uuid: "803f885391e24048af4bb96e664d37da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d15fc17020842d294fd650ba9c2d532"
I20260928 21:49:04.631323 4344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05ffe6cfa7e843df928ad57e90ae4b36" candidate_uuid: "803f885391e24048af4bb96e664d37da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900f4a9cf36e4af5aba0e56e35f910ab"
I20260928 21:49:04.631402 4484 raft_consensus.cc:3037] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:04.631402 4344 raft_consensus.cc:3037] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:04.632220 4344 raft_consensus.cc:2454] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 803f885391e24048af4bb96e664d37da in term 1.
I20260928 21:49:04.632222 4484 raft_consensus.cc:2454] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 803f885391e24048af4bb96e664d37da in term 1.
I20260928 21:49:04.632368 4558 leader_election.cc:302] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 803f885391e24048af4bb96e664d37da, 900f4a9cf36e4af5aba0e56e35f910ab; no voters:
I20260928 21:49:04.632486 4694 raft_consensus.cc:2781] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:04.632632 4694 raft_consensus.cc:686] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 1 LEADER]: Becoming Leader. State: Replica: 803f885391e24048af4bb96e664d37da, State: Running, Role: LEADER
I20260928 21:49:04.632716 4694 consensus_queue.cc:237] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } }
I20260928 21:49:04.633504 4186 catalog_manager.cc:5495] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da reported cstate change: term changed from 0 to 1, leader changed from <none> to 803f885391e24048af4bb96e664d37da (127.3.0.195). New cstate: current_term: 1 leader_uuid: "803f885391e24048af4bb96e664d37da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } health_report { overall_health: UNKNOWN } } }
I20260928 21:49:04.639954 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.637672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.3.0.195:33325)
0928 21:49:04.637710 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.637712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.637757 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:49:04.637839 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.637840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.637857 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.637911 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:04.638025 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.638474 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.639217 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.639411 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.639464 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.639645 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:04.639734 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:04.639804 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.639852 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.639873 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:49:04.641777 4344 raft_consensus.cc:1261] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab [term 1 FOLLOWER]: Refusing update from remote peer 803f885391e24048af4bb96e664d37da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:04.641803 4484 raft_consensus.cc:1261] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532 [term 1 FOLLOWER]: Refusing update from remote peer 803f885391e24048af4bb96e664d37da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:04.642099 4700 consensus_queue.cc:1035] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:04.642228 4694 consensus_queue.cc:1035] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [LEADER]: Connected to new peer: Peer: permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:04.647472 4186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:04.647552 4186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:04.648232 4186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7: Sending DeleteTablet for 3 replicas of tablet 05ffe6cfa7e843df928ad57e90ae4b36
I20260928 21:49:04.648572 4313 tablet_service.cc:1514] Processing DeleteTablet for tablet 05ffe6cfa7e843df928ad57e90ae4b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:04 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:53698
I20260928 21:49:04.648586 4604 tablet_service.cc:1514] Processing DeleteTablet for tablet 05ffe6cfa7e843df928ad57e90ae4b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:04 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:42546
I20260928 21:49:04.648767 4464 tablet_service.cc:1514] Processing DeleteTablet for tablet 05ffe6cfa7e843df928ad57e90ae4b36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:04 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:34160
I20260928 21:49:04.648834 4186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:04.648963 4186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:04.648983 4713 tablet_replica.cc:330] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: stopping tablet replica
I20260928 21:49:04.649066 4713 raft_consensus.cc:2229] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:04.649178 4713 raft_consensus.cc:2258] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:04.649180 4715 tablet_replica.cc:330] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: stopping tablet replica
I20260928 21:49:04.649326 4714 tablet_replica.cc:330] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: stopping tablet replica
I20260928 21:49:04.649343 4715 raft_consensus.cc:2229] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:04.649417 4714 raft_consensus.cc:2229] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:04.649521 4714 raft_consensus.cc:2258] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:04.649549 4715 raft_consensus.cc:2258] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:04.650074 4714 ts_tablet_manager.cc:1905] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:04.650127 4313 tablet_service.cc:1467] Processing CreateTablet for tablet 38d4ece2529d4cce9b7106917840761a (DEFAULT_TABLE table=test-table [id=4464905a8f3a4a9280f00a8ba7a28a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:04.650220 4313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d4ece2529d4cce9b7106917840761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:04.650461 4713 ts_tablet_manager.cc:1905] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:04.650591 4715 ts_tablet_manager.cc:1905] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:04.651753 4464 tablet_service.cc:1467] Processing CreateTablet for tablet 38d4ece2529d4cce9b7106917840761a (DEFAULT_TABLE table=test-table [id=4464905a8f3a4a9280f00a8ba7a28a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:04.651857 4464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d4ece2529d4cce9b7106917840761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:04.652149 4714 ts_tablet_manager.cc:1918] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:04.652218 4714 log.cc:1196] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/05ffe6cfa7e843df928ad57e90ae4b36
I20260928 21:49:04.652472 4714 ts_tablet_manager.cc:1939] T 05ffe6cfa7e843df928ad57e90ae4b36 P 803f885391e24048af4bb96e664d37da: Deleting consensus metadata
I20260928 21:49:04.652518 4691 tablet_bootstrap.cc:492] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: Bootstrap starting.
I20260928 21:49:04.652900 4691 tablet_bootstrap.cc:654] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:04.653415 4171 catalog_manager.cc:4841] TS 803f885391e24048af4bb96e664d37da (127.3.0.195:33325): tablet 05ffe6cfa7e843df928ad57e90ae4b36 (table test-table [id=ab7882fb522e47aab4e7b90bec33b989]) successfully deleted
I20260928 21:49:04.653445 4691 tablet_bootstrap.cc:492] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: No bootstrap required, opened a new log
I20260928 21:49:04.653478 4691 ts_tablet_manager.cc:1397] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:04.653491 4713 ts_tablet_manager.cc:1918] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:04.653556 4713 log.cc:1196] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/05ffe6cfa7e843df928ad57e90ae4b36
I20260928 21:49:04.653601 4604 tablet_service.cc:1467] Processing CreateTablet for tablet 38d4ece2529d4cce9b7106917840761a (DEFAULT_TABLE table=test-table [id=4464905a8f3a4a9280f00a8ba7a28a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:04.653609 4691 raft_consensus.cc:348] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.653667 4691 raft_consensus.cc:374] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:04.653685 4691 raft_consensus.cc:729] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900f4a9cf36e4af5aba0e56e35f910ab, State: Initialized, Role: FOLLOWER
I20260928 21:49:04.653687 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d4ece2529d4cce9b7106917840761a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:04.653724 4691 consensus_queue.cc:260] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.653784 4713 ts_tablet_manager.cc:1939] T 05ffe6cfa7e843df928ad57e90ae4b36 P 900f4a9cf36e4af5aba0e56e35f910ab: Deleting consensus metadata
I20260928 21:49:04.653810 4691 ts_tablet_manager.cc:1428] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.654311 4174 catalog_manager.cc:4841] TS 900f4a9cf36e4af5aba0e56e35f910ab (127.3.0.193:45151): tablet 05ffe6cfa7e843df928ad57e90ae4b36 (table test-table [id=ab7882fb522e47aab4e7b90bec33b989]) successfully deleted
I20260928 21:49:04.654840 4690 tablet_bootstrap.cc:492] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: Bootstrap starting.
I20260928 21:49:04.655231 4690 tablet_bootstrap.cc:654] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:04.655622 4695 tablet_bootstrap.cc:492] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: Bootstrap starting.
I20260928 21:49:04.655988 4695 tablet_bootstrap.cc:654] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:04.656387 4690 tablet_bootstrap.cc:492] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: No bootstrap required, opened a new log
I20260928 21:49:04.656432 4690 ts_tablet_manager.cc:1397] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:04.656567 4695 tablet_bootstrap.cc:492] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: No bootstrap required, opened a new log
I20260928 21:49:04.656599 4695 ts_tablet_manager.cc:1397] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:04.656589 4690 raft_consensus.cc:348] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.656656 4690 raft_consensus.cc:374] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:04.656677 4690 raft_consensus.cc:729] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803f885391e24048af4bb96e664d37da, State: Initialized, Role: FOLLOWER
I20260928 21:49:04.656714 4690 consensus_queue.cc:260] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.656705 4695 raft_consensus.cc:348] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.656769 4695 raft_consensus.cc:374] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:04.656785 4695 raft_consensus.cc:729] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d15fc17020842d294fd650ba9c2d532, State: Initialized, Role: FOLLOWER
I20260928 21:49:04.656806 4690 ts_tablet_manager.cc:1428] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.656853 4695 consensus_queue.cc:260] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.656944 4695 ts_tablet_manager.cc:1428] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.657845 4715 ts_tablet_manager.cc:1918] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:04.657918 4715 log.cc:1196] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/05ffe6cfa7e843df928ad57e90ae4b36
I20260928 21:49:04.658147 4715 ts_tablet_manager.cc:1939] T 05ffe6cfa7e843df928ad57e90ae4b36 P 4d15fc17020842d294fd650ba9c2d532: Deleting consensus metadata
W20260928 21:49:04.658411 4531 tablet.cc:2369] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:49:04.658571 4172 catalog_manager.cc:4841] TS 4d15fc17020842d294fd650ba9c2d532 (127.3.0.194:45937): tablet 05ffe6cfa7e843df928ad57e90ae4b36 (table test-table [id=ab7882fb522e47aab4e7b90bec33b989]) successfully deleted
W20260928 21:49:04.750764 4391 tablet.cc:2369] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:49:04.785297 4708 raft_consensus.cc:482] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:04.785398 4708 raft_consensus.cc:504] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.785567 4708 leader_election.cc:288] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900f4a9cf36e4af5aba0e56e35f910ab (127.3.0.193:45151), 4d15fc17020842d294fd650ba9c2d532 (127.3.0.194:45937)
I20260928 21:49:04.785786 4344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d4ece2529d4cce9b7106917840761a" candidate_uuid: "803f885391e24048af4bb96e664d37da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" is_pre_election: true
I20260928 21:49:04.785825 4484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d4ece2529d4cce9b7106917840761a" candidate_uuid: "803f885391e24048af4bb96e664d37da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d15fc17020842d294fd650ba9c2d532" is_pre_election: true
I20260928 21:49:04.785892 4344 raft_consensus.cc:2454] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 803f885391e24048af4bb96e664d37da in term 0.
I20260928 21:49:04.785912 4484 raft_consensus.cc:2454] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 803f885391e24048af4bb96e664d37da in term 0.
I20260928 21:49:04.786046 4556 leader_election.cc:302] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d15fc17020842d294fd650ba9c2d532, 803f885391e24048af4bb96e664d37da; no voters:
I20260928 21:49:04.786118 4708 raft_consensus.cc:2781] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:04.786140 4708 raft_consensus.cc:482] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:04.786158 4708 raft_consensus.cc:3037] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:04.787057 4708 raft_consensus.cc:504] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.787197 4708 leader_election.cc:288] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [CANDIDATE]: Term 1 election: Requested vote from peers 900f4a9cf36e4af5aba0e56e35f910ab (127.3.0.193:45151), 4d15fc17020842d294fd650ba9c2d532 (127.3.0.194:45937)
I20260928 21:49:04.787384 4344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d4ece2529d4cce9b7106917840761a" candidate_uuid: "803f885391e24048af4bb96e664d37da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900f4a9cf36e4af5aba0e56e35f910ab"
I20260928 21:49:04.787384 4484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d4ece2529d4cce9b7106917840761a" candidate_uuid: "803f885391e24048af4bb96e664d37da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d15fc17020842d294fd650ba9c2d532"
I20260928 21:49:04.787454 4344 raft_consensus.cc:3037] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:04.787456 4484 raft_consensus.cc:3037] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:04.788170 4344 raft_consensus.cc:2454] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 803f885391e24048af4bb96e664d37da in term 1.
I20260928 21:49:04.788245 4484 raft_consensus.cc:2454] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 803f885391e24048af4bb96e664d37da in term 1.
I20260928 21:49:04.788409 4558 leader_election.cc:302] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 803f885391e24048af4bb96e664d37da, 900f4a9cf36e4af5aba0e56e35f910ab; no voters:
I20260928 21:49:04.788565 4708 raft_consensus.cc:2781] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:04.788647 4708 raft_consensus.cc:686] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 1 LEADER]: Becoming Leader. State: Replica: 803f885391e24048af4bb96e664d37da, State: Running, Role: LEADER
I20260928 21:49:04.788686 4708 consensus_queue.cc:237] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.789189 4187 catalog_manager.cc:5495] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da reported cstate change: term changed from 0 to 1, leader changed from <none> to 803f885391e24048af4bb96e664d37da (127.3.0.195). New cstate: current_term: 1 leader_uuid: "803f885391e24048af4bb96e664d37da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } health_report { overall_health: HEALTHY } } }
W20260928 21:49:04.802699 4671 tablet.cc:2369] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:04.827337 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.821202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33975 (local address 127.0.0.1:39094)
0928 21:49:04.821352 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.821369 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.821438 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.821556 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.821557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.821850 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:04.821896 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.821899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.822525 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.822525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.823059 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.823060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.823368 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.823369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.823417 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.823891 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:04.823896 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:04.825236 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:04.826430 (+ 1194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.826436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.826437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.826618 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.826772 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:04.826773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:04.826773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:04.826831 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:04.827072 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:04.827077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:04.827122 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.827175 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.827266 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260928 21:49:04.827436 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.821236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.3.0.254:33975)
0928 21:49:04.821291 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.821292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.821434 (+ 142us) server_negotiation.cc:407] Connection header received
0928 21:49:04.821490 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.821491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.821501 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.821548 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.821973 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.822458 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.823133 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.823306 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.823359 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.825316 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.825317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.825318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.825324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.826365 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.826672 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.826672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.826673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.826714 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.826894 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.826895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.826895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.827022 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.827072 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.827195 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.827361 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":30}
I20260928 21:49:04.829191 4187 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:39094:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:49:04.829373 4187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:04.830556 4604 tablet_service.cc:1467] Processing CreateTablet for tablet 96d24b88caeb4dae8140abae95c43d74 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=027a26e2bd1341b7a916c5301c5a414e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:49:04.830574 4464 tablet_service.cc:1467] Processing CreateTablet for tablet 96d24b88caeb4dae8140abae95c43d74 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=027a26e2bd1341b7a916c5301c5a414e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:49:04.830574 4313 tablet_service.cc:1467] Processing CreateTablet for tablet 96d24b88caeb4dae8140abae95c43d74 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=027a26e2bd1341b7a916c5301c5a414e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:49:04.830698 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d24b88caeb4dae8140abae95c43d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:04.830708 4313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d24b88caeb4dae8140abae95c43d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:04.830709 4464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d24b88caeb4dae8140abae95c43d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:04.831715 4690 tablet_bootstrap.cc:492] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: Bootstrap starting.
I20260928 21:49:04.832017 4690 tablet_bootstrap.cc:654] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:04.832114 4695 tablet_bootstrap.cc:492] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: Bootstrap starting.
I20260928 21:49:04.832556 4695 tablet_bootstrap.cc:654] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:04.832589 4690 tablet_bootstrap.cc:492] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: No bootstrap required, opened a new log
I20260928 21:49:04.832641 4690 ts_tablet_manager.cc:1397] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:04.832749 4690 raft_consensus.cc:348] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.832810 4690 raft_consensus.cc:374] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:04.832826 4690 raft_consensus.cc:729] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803f885391e24048af4bb96e664d37da, State: Initialized, Role: FOLLOWER
I20260928 21:49:04.832871 4691 tablet_bootstrap.cc:492] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: Bootstrap starting.
I20260928 21:49:04.832865 4690 consensus_queue.cc:260] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.832958 4690 ts_tablet_manager.cc:1428] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.832976 4708 tablet_replica.cc:439] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } }
I20260928 21:49:04.833065 4708 tablet_replica.cc:442] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:04.833258 4691 tablet_bootstrap.cc:654] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:04.833675 4695 tablet_bootstrap.cc:492] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: No bootstrap required, opened a new log
I20260928 21:49:04.833730 4695 ts_tablet_manager.cc:1397] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:49:04.833876 4691 tablet_bootstrap.cc:492] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: No bootstrap required, opened a new log
I20260928 21:49:04.833873 4695 raft_consensus.cc:348] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.833917 4695 raft_consensus.cc:374] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:04.833918 4691 ts_tablet_manager.cc:1397] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:04.833930 4695 raft_consensus.cc:729] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d15fc17020842d294fd650ba9c2d532, State: Initialized, Role: FOLLOWER
I20260928 21:49:04.833969 4695 consensus_queue.cc:260] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.834051 4695 ts_tablet_manager.cc:1428] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.834048 4691 raft_consensus.cc:348] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.834115 4691 raft_consensus.cc:374] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:04.834141 4691 raft_consensus.cc:729] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900f4a9cf36e4af5aba0e56e35f910ab, State: Initialized, Role: FOLLOWER
I20260928 21:49:04.834183 4691 consensus_queue.cc:260] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.834177 4698 tablet_replica.cc:439] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } }
I20260928 21:49:04.834242 4698 tablet_replica.cc:442] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:04.834265 4691 ts_tablet_manager.cc:1428] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:04.834336 4697 tablet_replica.cc:439] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } }
I20260928 21:49:04.834399 4697 tablet_replica.cc:442] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:04.908783 4698 raft_consensus.cc:482] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:04.908870 4698 raft_consensus.cc:504] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.909153 4698 leader_election.cc:288] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900f4a9cf36e4af5aba0e56e35f910ab (127.3.0.193:45151), 803f885391e24048af4bb96e664d37da (127.3.0.195:33325)
I20260928 21:49:04.912578 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.909306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:33325 (local address 127.3.0.194:50999)
0928 21:49:04.909355 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.909378 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.909429 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.909569 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.909570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.909571 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:04.909666 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.909668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.910559 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.910560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.911843 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.911847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.912352 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.912352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.912415 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.912418 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.912484 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.912524 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":5}
I20260928 21:49:04.912587 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.909406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:50999 (local address 127.3.0.195:33325)
0928 21:49:04.909454 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.909456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.909465 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:49:04.909495 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.909496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.909504 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.909558 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:04.909720 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.910504 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.911918 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.912276 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.912341 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.912353 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.912487 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.912521 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:49:04.912806 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d24b88caeb4dae8140abae95c43d74" candidate_uuid: "4d15fc17020842d294fd650ba9c2d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "803f885391e24048af4bb96e664d37da" is_pre_election: true
I20260928 21:49:04.912945 4624 raft_consensus.cc:2454] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d15fc17020842d294fd650ba9c2d532 in term 0.
I20260928 21:49:04.913163 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.909419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:55471 (local address 127.3.0.193:45151)
0928 21:49:04.909473 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.909475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.909564 (+ 89us) server_negotiation.cc:407] Connection header received
0928 21:49:04.909598 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.909599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.909607 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.909656 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:04.909791 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.910487 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.912352 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.912846 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.912907 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.912917 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.913059 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.913112 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:49:04.913167 4729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.909342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:45151 (local address 127.3.0.194:55471)
0928 21:49:04.909497 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.909506 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.909556 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.909665 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.909666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.909667 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:04.909727 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.909728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.910559 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.910560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.912267 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.912270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.912918 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.912919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.912994 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.912998 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.913049 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.913088 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":59,"thread_start_us":64,"threads_started":1}
I20260928 21:49:04.913159 4415 leader_election.cc:302] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d15fc17020842d294fd650ba9c2d532, 803f885391e24048af4bb96e664d37da; no voters:
I20260928 21:49:04.913278 4698 raft_consensus.cc:2781] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:04.913304 4698 raft_consensus.cc:482] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:04.913321 4698 raft_consensus.cc:3037] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:04.913436 4344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d24b88caeb4dae8140abae95c43d74" candidate_uuid: "4d15fc17020842d294fd650ba9c2d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" is_pre_election: true
I20260928 21:49:04.913519 4344 raft_consensus.cc:2454] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d15fc17020842d294fd650ba9c2d532 in term 0.
I20260928 21:49:04.914069 4698 raft_consensus.cc:504] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.914187 4698 leader_election.cc:288] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [CANDIDATE]: Term 1 election: Requested vote from peers 900f4a9cf36e4af5aba0e56e35f910ab (127.3.0.193:45151), 803f885391e24048af4bb96e664d37da (127.3.0.195:33325)
I20260928 21:49:04.914310 4344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d24b88caeb4dae8140abae95c43d74" candidate_uuid: "4d15fc17020842d294fd650ba9c2d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900f4a9cf36e4af5aba0e56e35f910ab"
I20260928 21:49:04.914323 4624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d24b88caeb4dae8140abae95c43d74" candidate_uuid: "4d15fc17020842d294fd650ba9c2d532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "803f885391e24048af4bb96e664d37da"
I20260928 21:49:04.914382 4624 raft_consensus.cc:3037] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:04.914382 4344 raft_consensus.cc:3037] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:04.914956 4624 raft_consensus.cc:2454] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d15fc17020842d294fd650ba9c2d532 in term 1.
I20260928 21:49:04.914970 4344 raft_consensus.cc:2454] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d15fc17020842d294fd650ba9c2d532 in term 1.
I20260928 21:49:04.915110 4418 leader_election.cc:302] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d15fc17020842d294fd650ba9c2d532, 900f4a9cf36e4af5aba0e56e35f910ab; no voters:
I20260928 21:49:04.915202 4698 raft_consensus.cc:2781] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:04.915336 4698 raft_consensus.cc:686] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [term 1 LEADER]: Becoming Leader. State: Replica: 4d15fc17020842d294fd650ba9c2d532, State: Running, Role: LEADER
I20260928 21:49:04.915403 4698 consensus_queue.cc:237] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } }
I20260928 21:49:04.915706 4730 tablet_replica.cc:439] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: TxnStatusTablet state changed. Reason: New leader 4d15fc17020842d294fd650ba9c2d532. Latest consensus state: current_term: 1 leader_uuid: "4d15fc17020842d294fd650ba9c2d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } }
I20260928 21:49:04.915846 4730 tablet_replica.cc:442] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:49:04.916026 4732 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:49:04.916136 4732 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:49:04.916066 4187 catalog_manager.cc:5495] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d15fc17020842d294fd650ba9c2d532 (127.3.0.194). New cstate: current_term: 1 leader_uuid: "4d15fc17020842d294fd650ba9c2d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } health_report { overall_health: UNKNOWN } } }
I20260928 21:49:04.927276 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.925066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.3.0.194:45937)
0928 21:49:04.925109 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.925111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.925133 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:49:04.925224 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.925225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.925243 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.925295 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:04.925425 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.925879 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.926593 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.926775 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.926827 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.926988 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:04.927081 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:04.927142 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.927189 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.927209 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:04.927301 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.925008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:45937 (local address 127.0.0.1:34176)
0928 21:49:04.925088 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.925100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.925153 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.925309 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.925310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.925311 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:04.925367 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.925369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.925949 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.925950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.926518 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.926519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.926838 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.926839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.926901 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.926915 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:04.927138 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:04.927145 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.927188 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.927236 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 21:49:04.928280 4344 raft_consensus.cc:1261] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab [term 1 FOLLOWER]: Refusing update from remote peer 4d15fc17020842d294fd650ba9c2d532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:04.928325 4624 raft_consensus.cc:1261] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da [term 1 FOLLOWER]: Refusing update from remote peer 4d15fc17020842d294fd650ba9c2d532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:04.928535 4698 consensus_queue.cc:1035] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:04.928654 4730 consensus_queue.cc:1035] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:04.929109 4697 tablet_replica.cc:439] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: TxnStatusTablet state changed. Reason: New leader 4d15fc17020842d294fd650ba9c2d532. Latest consensus state: current_term: 1 leader_uuid: "4d15fc17020842d294fd650ba9c2d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } }
I20260928 21:49:04.929203 4697 tablet_replica.cc:442] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:04.929283 4708 tablet_replica.cc:439] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: TxnStatusTablet state changed. Reason: New leader 4d15fc17020842d294fd650ba9c2d532. Latest consensus state: current_term: 1 leader_uuid: "4d15fc17020842d294fd650ba9c2d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } }
I20260928 21:49:04.929381 4708 tablet_replica.cc:442] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:04.929620 4698 tablet_replica.cc:439] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d15fc17020842d294fd650ba9c2d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } }
I20260928 21:49:04.929700 4698 tablet_replica.cc:442] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:49:04.930148 4705 mvcc.cc:205] Tried to move back new op lower bound from 7334429265624584192 to 7334429265574080512. Current Snapshot: MvccSnapshot[applied={T|T < 7334429265624584192}]
I20260928 21:49:04.930253 4697 tablet_replica.cc:439] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d15fc17020842d294fd650ba9c2d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } }
I20260928 21:49:04.930323 4697 tablet_replica.cc:442] T 96d24b88caeb4dae8140abae95c43d74 P 900f4a9cf36e4af5aba0e56e35f910ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:04.930399 4704 mvcc.cc:205] Tried to move back new op lower bound from 7334429265624584192 to 7334429265574080512. Current Snapshot: MvccSnapshot[applied={T|T < 7334429265624584192 or (T in {7334429265624584192})}]
I20260928 21:49:04.930931 4730 tablet_replica.cc:439] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d15fc17020842d294fd650ba9c2d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } }
I20260928 21:49:04.931005 4730 tablet_replica.cc:442] T 96d24b88caeb4dae8140abae95c43d74 P 4d15fc17020842d294fd650ba9c2d532: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:49:04.931255 4703 mvcc.cc:205] Tried to move back new op lower bound from 7334429265624584192 to 7334429265574080512. Current Snapshot: MvccSnapshot[applied={T|T < 7334429265624584192 or (T in {7334429265624584192})}]
I20260928 21:49:04.931458 4708 tablet_replica.cc:439] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d15fc17020842d294fd650ba9c2d532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 } } peers { permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 } } peers { permanent_uuid: "803f885391e24048af4bb96e664d37da" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 33325 } } }
I20260928 21:49:04.931543 4708 tablet_replica.cc:442] T 96d24b88caeb4dae8140abae95c43d74 P 803f885391e24048af4bb96e664d37da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:04.935904 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.933965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:39409 (local address 127.3.0.194:45937)
0928 21:49:04.934011 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.934012 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.934019 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:49:04.934035 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.934035 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.934042 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.934084 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:04.934182 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.934629 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.935330 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.935513 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.935565 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.935705 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:04.935734 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:04.935773 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.935828 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.935844 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:49:04.935948 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.933787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:45937 (local address 127.3.0.195:39409)
0928 21:49:04.933855 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.933871 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.933913 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.934094 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.934095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.934097 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:04.934138 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.934139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.934695 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.934696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.935261 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.935263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.935568 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.935570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.935627 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.935641 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:04.935779 (+ 138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:04.935780 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.935816 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.935872 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 21:49:04.937963 4484 raft_consensus.cc:1261] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532 [term 1 FOLLOWER]: Refusing update from remote peer 803f885391e24048af4bb96e664d37da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:04.937965 4344 raft_consensus.cc:1261] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab [term 1 FOLLOWER]: Refusing update from remote peer 803f885391e24048af4bb96e664d37da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:04.938162 4708 consensus_queue.cc:1035] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [LEADER]: Connected to new peer: Peer: permanent_uuid: "900f4a9cf36e4af5aba0e56e35f910ab" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:04.938263 4694 consensus_queue.cc:1035] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d15fc17020842d294fd650ba9c2d532" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 45937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:04.945537 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.943035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:57883 (local address 127.3.0.195:33325)
0928 21:49:04.943070 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.943072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.943194 (+ 122us) server_negotiation.cc:407] Connection header received
0928 21:49:04.943258 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.943258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.943267 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.943303 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:04.943444 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.943876 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.944983 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.945159 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.945207 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.945377 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:04.945399 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:04.945429 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.945479 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.945493 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:49:04.945605 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.943001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:33325 (local address 127.3.0.194:57883)
0928 21:49:04.943145 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:04.943155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:04.943207 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:04.943324 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:04.943325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:04.943338 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:04.943389 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.943390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.943935 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.943936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.944912 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:04.944915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.945220 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.945221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:04.945284 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.945306 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:04.945432 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:04.945433 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:49:04.945477 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:49:04.945515 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260928 21:49:04.954492 4187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:04.954586 4187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:04.955142 4187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d96b4530e23943d19996ae3bcf60ecc7: Sending DeleteTablet for 3 replicas of tablet 38d4ece2529d4cce9b7106917840761a
I20260928 21:49:04.955372 4313 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d4ece2529d4cce9b7106917840761a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:04 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:53698
I20260928 21:49:04.955477 4713 tablet_replica.cc:330] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: stopping tablet replica
I20260928 21:49:04.955523 4713 raft_consensus.cc:2229] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:04.955571 4713 raft_consensus.cc:2258] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:04.955706 4604 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d4ece2529d4cce9b7106917840761a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:04 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:42546
I20260928 21:49:04.955791 4714 tablet_replica.cc:330] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: stopping tablet replica
I20260928 21:49:04.955840 4714 raft_consensus.cc:2229] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:04.955890 4714 raft_consensus.cc:2258] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:04.955963 4464 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d4ece2529d4cce9b7106917840761a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:04 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:34160
I20260928 21:49:04.956058 4715 tablet_replica.cc:330] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: stopping tablet replica
I20260928 21:49:04.956108 4715 raft_consensus.cc:2229] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:04.956142 4715 raft_consensus.cc:2258] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:04.956429 4715 ts_tablet_manager.cc:1905] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:04.956493 4714 ts_tablet_manager.cc:1905] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:04.957693 4713 ts_tablet_manager.cc:1905] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:04.957926 4714 ts_tablet_manager.cc:1918] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:04.957947 4715 ts_tablet_manager.cc:1918] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:04.957971 4714 log.cc:1196] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/38d4ece2529d4cce9b7106917840761a
I20260928 21:49:04.957995 4715 log.cc:1196] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/38d4ece2529d4cce9b7106917840761a
I20260928 21:49:04.958156 4714 ts_tablet_manager.cc:1939] T 38d4ece2529d4cce9b7106917840761a P 803f885391e24048af4bb96e664d37da: Deleting consensus metadata
I20260928 21:49:04.958192 4715 ts_tablet_manager.cc:1939] T 38d4ece2529d4cce9b7106917840761a P 4d15fc17020842d294fd650ba9c2d532: Deleting consensus metadata
I20260928 21:49:04.958483 4172 catalog_manager.cc:4841] TS 4d15fc17020842d294fd650ba9c2d532 (127.3.0.194:45937): tablet 38d4ece2529d4cce9b7106917840761a (table test-table [id=4464905a8f3a4a9280f00a8ba7a28a78]) successfully deleted
I20260928 21:49:04.958573 4171 catalog_manager.cc:4841] TS 803f885391e24048af4bb96e664d37da (127.3.0.195:33325): tablet 38d4ece2529d4cce9b7106917840761a (table test-table [id=4464905a8f3a4a9280f00a8ba7a28a78]) successfully deleted
I20260928 21:49:04.959015 4713 ts_tablet_manager.cc:1918] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:04.959062 4713 log.cc:1196] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/38d4ece2529d4cce9b7106917840761a
I20260928 21:49:04.959219 4713 ts_tablet_manager.cc:1939] T 38d4ece2529d4cce9b7106917840761a P 900f4a9cf36e4af5aba0e56e35f910ab: Deleting consensus metadata
I20260928 21:49:04.959525 4174 catalog_manager.cc:4841] TS 900f4a9cf36e4af5aba0e56e35f910ab (127.3.0.193:45151): tablet 38d4ece2529d4cce9b7106917840761a (table test-table [id=4464905a8f3a4a9280f00a8ba7a28a78]) successfully deleted
Sep 28 21:49:04 dist-test-slave-jrck krb5kdc[4142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632144, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:49:04.968441 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.956217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.3.0.193:45151)
0928 21:49:04.956335 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.956337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.960622 (+ 4285us) server_negotiation.cc:407] Connection header received
0928 21:49:04.960687 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.960688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.960696 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.960739 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.961373 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.961823 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.962854 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.963031 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.963074 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.966050 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.966059 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.966059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.966065 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.967455 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.967748 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.967752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.967752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.967803 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.967962 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.967963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.967963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.968097 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.968239 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.968277 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.968352 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260928 21:49:04.968752 4381 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53712
I20260928 21:49:04.975275 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:04.969665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.3.0.254:33975)
0928 21:49:04.969721 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:04.969723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:04.969731 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:04.969753 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:04.969754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:04.969764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:04.969835 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:04.970258 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.971018 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.971635 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:04.971936 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:04.972000 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:04.973030 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:04.973031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:04.973032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:04.973039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:04.974380 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.974653 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.974654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.974654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.974714 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:04.974843 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:04.974844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:04.974844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:04.974965 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:04.975056 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:04.975118 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:49:04.975187 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 21:49:04.975459 4187 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39096
I20260928 21:49:04.976073 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 4259
I20260928 21:49:04.981520 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 4399
I20260928 21:49:04.985802 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 4539
I20260928 21:49:04.990288 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 4155
2026-09-28T21:49:04Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2104 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:05 dist-test-slave-jrck krb5kdc[4750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:05 dist-test-slave-jrck krb5kdc[4750](info): set up 2 sockets
Sep 28 21:49:05 dist-test-slave-jrck krb5kdc[4750](info): commencing operation
krb5kdc: starting...
W20260928 21:49:05.935946 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:06.042531 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:44633
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:06.122430 4763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:06.122625 4763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:06.122653 4763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:06.123755 4763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:06.123793 4763 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:49:06.123811 4763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:06.123826 4763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:06.123842 4763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:06.123857 4763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:06.125334 4763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:44633
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:06.125571 4763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:06.125762 4763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:06.130162 4768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:06.130152 4771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:06.130215 4763 server_base.cc:1034] running on GCE node
W20260928 21:49:06.130167 4769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:06.130594 4763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:06.130859 4763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:06.131994 4763 hybrid_clock.cc:648] HybridClock initialized: now 1790632146131978 us; error 32 us; skew 500 ppm
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:06.134176 4763 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:49:06.134548 4763 webserver.cc:460] Webserver started at http://127.3.0.254:42971/ using document root <none> and password file <none>
I20260928 21:49:06.134729 4763 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:06.134774 4763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:06.134902 4763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:06.135893 4763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "cc72d04ef78041fab55cab014157b2f8"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "412ed1e684655377091eea587b03b11e"
server_key_iv: "fd7834fabbecdde869621de9db330546"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.136200 4763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "cc72d04ef78041fab55cab014157b2f8"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "412ed1e684655377091eea587b03b11e"
server_key_iv: "fd7834fabbecdde869621de9db330546"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.137420 4763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:49:06.138046 4778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:06.138186 4763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:06.138262 4763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "cc72d04ef78041fab55cab014157b2f8"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "412ed1e684655377091eea587b03b11e"
server_key_iv: "fd7834fabbecdde869621de9db330546"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.138326 4763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:06.147675 4763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:06.149272 4763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:06.149493 4763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:06.153630 4763 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:44633
I20260928 21:49:06.153698 4840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:44633 every 8 connection(s)
I20260928 21:49:06.153963 4763 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:06.154553 4841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:06.156613 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 4763
I20260928 21:49:06.156703 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:06.156857 3075 external_mini_cluster.cc:1428] Setting key 6b04fbccae4f795d2334c07251299b34
I20260928 21:49:06.156908 4841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8: Bootstrap starting.
I20260928 21:49:06.157536 4841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:06.157785 4841 log.cc:824] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:06.158493 4841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8: No bootstrap required, opened a new log
I20260928 21:49:06.160678 4841 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc72d04ef78041fab55cab014157b2f8" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44633 } }
I20260928 21:49:06.160853 4841 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:06.160887 4841 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc72d04ef78041fab55cab014157b2f8, State: Initialized, Role: FOLLOWER
I20260928 21:49:06.161007 4841 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc72d04ef78041fab55cab014157b2f8" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44633 } }
I20260928 21:49:06.161064 4841 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:06.161094 4841 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:06.161137 4841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:06.161698 4841 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc72d04ef78041fab55cab014157b2f8" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44633 } }
I20260928 21:49:06.161791 4841 leader_election.cc:302] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc72d04ef78041fab55cab014157b2f8; no voters:
I20260928 21:49:06.161949 4841 leader_election.cc:288] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:06.162016 4846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:06.162163 4846 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [term 1 LEADER]: Becoming Leader. State: Replica: cc72d04ef78041fab55cab014157b2f8, State: Running, Role: LEADER
I20260928 21:49:06.162194 4841 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:06.162256 4846 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc72d04ef78041fab55cab014157b2f8" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44633 } }
I20260928 21:49:06.162531 4846 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc72d04ef78041fab55cab014157b2f8. Latest consensus state: current_term: 1 leader_uuid: "cc72d04ef78041fab55cab014157b2f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc72d04ef78041fab55cab014157b2f8" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44633 } } }
I20260928 21:49:06.162616 4846 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:06.162541 4848 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc72d04ef78041fab55cab014157b2f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc72d04ef78041fab55cab014157b2f8" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44633 } } }
I20260928 21:49:06.162745 4848 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:06.162858 4852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:06.163328 4852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:06.166917 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.157314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.3.0.254:44633)
0928 21:49:06.157470 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.157479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.157504 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:49:06.157993 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.158002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.158064 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.158202 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:06.158885 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.159739 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.160801 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.161353 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.161431 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.164038 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:06.164043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:06.164043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:06.164054 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:06.165570 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.165949 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.165951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.165951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.166024 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.166199 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.166199 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.166199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.166397 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:06.166486 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.166626 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.166718 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260928 21:49:06.169126 4852 catalog_manager.cc:1347] Generated new cluster ID: 503e96a1ad6a42d3a85fda12ca8b529a
I20260928 21:49:06.169173 4852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:49:06.179666 4852 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:06.180199 4852 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:06.185204 4852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8: Generated new TSK 0
I20260928 21:49:06.185364 4852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:49:06.209069 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:44633
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:06.287468 4867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:06.287657 4867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:06.287680 4867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:06.288841 4867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:06.288883 4867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:06.288902 4867 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:49:06.288913 4867 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:49:06.288932 4867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:06.290365 4867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.0.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:06.290577 4867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:06.290777 4867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:06.295197 4873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:06.295208 4872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:06.295208 4875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:06.295666 4867 server_base.cc:1034] running on GCE node
I20260928 21:49:06.295826 4867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:06.296036 4867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:06.297173 4867 hybrid_clock.cc:648] HybridClock initialized: now 1790632146297156 us; error 26 us; skew 500 ppm
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:06.299393 4867 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:49:06.299789 4867 webserver.cc:460] Webserver started at http://127.3.0.193:38921/ using document root <none> and password file <none>
I20260928 21:49:06.299988 4867 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:06.300032 4867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:06.300154 4867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:06.301180 4867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "3a3ee5995204477fbf96843b3dfdc271"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "8920b486e275b8e076e548883b2eec52"
server_key_iv: "4769b0e6e91b934232a8f265183c3765"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.301496 4867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "3a3ee5995204477fbf96843b3dfdc271"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "8920b486e275b8e076e548883b2eec52"
server_key_iv: "4769b0e6e91b934232a8f265183c3765"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.302671 4867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:06.303310 4882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:06.303452 4867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:06.303527 4867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "3a3ee5995204477fbf96843b3dfdc271"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "8920b486e275b8e076e548883b2eec52"
server_key_iv: "4769b0e6e91b934232a8f265183c3765"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.303583 4867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:06.316695 4867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:06.318281 4867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:06.318413 4867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:06.318873 4867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:06.318912 4867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:06.318938 4867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:06.318962 4867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:06.325722 4867 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:38723
I20260928 21:49:06.326037 4867 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:49:06.326426 4997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:38723 every 8 connection(s)
I20260928 21:49:06.327764 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.319255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:38093 (local address 127.3.0.254:44633)
0928 21:49:06.319302 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.319304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.319516 (+ 212us) server_negotiation.cc:407] Connection header received
0928 21:49:06.319993 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.319995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.320007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.320068 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:06.320755 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.321515 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.322494 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.322803 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.322889 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.325305 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:06.325307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:06.325307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:06.325314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:06.326686 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.327077 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.327078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.327078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.327125 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.327297 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.327297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.327297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.327395 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:06.327439 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.327639 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.327702 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:49:06.327939 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.319257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44633 (local address 127.3.0.193:38093)
0928 21:49:06.319391 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.319424 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.319903 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.320132 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.320135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.320435 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:06.320675 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.320680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.321609 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.321610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.322423 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.322426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.322916 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.322917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.323045 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.323554 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:06.323565 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:06.325210 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:06.326750 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.326754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.326756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.327016 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.327169 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.327169 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.327169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.327257 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.327436 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:06.327439 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:06.327545 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.327621 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.327733 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260928 21:49:06.328399 4998 heartbeater.cc:347] Connected to a master server at 127.3.0.254:44633
I20260928 21:49:06.328500 4998 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:06.328671 4998 heartbeater.cc:511] Master 127.3.0.254:44633 requested a full tablet report, sending...
I20260928 21:49:06.329128 4795 ts_manager.cc:194] Registered new tserver with Master: 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723)
I20260928 21:49:06.329756 4795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:38093
I20260928 21:49:06.333551 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 4867
I20260928 21:49:06.333647 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:06.333856 3075 external_mini_cluster.cc:1428] Setting key a30a9eacc85f92ca5ccf62a21104c678
I20260928 21:49:06.334041 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.329326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44633 (local address 127.3.0.193:43417)
0928 21:49:06.329481 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.329491 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.329541 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.329648 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.329649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.329805 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:06.329894 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.329896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.330535 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.330536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.331027 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.331029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.331307 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.331308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.331355 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.331702 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:06.331707 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:06.331987 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:06.333142 (+ 1155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.333142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.333142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.333285 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.333433 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.333433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.333434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.333477 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.333736 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:06.333737 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:06.333794 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.333859 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.333951 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260928 21:49:06.334052 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.329393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:43417 (local address 127.3.0.254:44633)
0928 21:49:06.329432 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.329433 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.329541 (+ 108us) server_negotiation.cc:407] Connection header received
0928 21:49:06.329593 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.329593 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.329603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.329646 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:06.329953 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.330450 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.331087 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.331258 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.331303 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.332046 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:06.332046 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:06.332047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:06.332051 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:06.333082 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.333337 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.333338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.333338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.333383 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.333527 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.333527 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.333527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.333647 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:06.333706 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.333897 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.333974 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:49:06.355381 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:44633
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:06.433950 5007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:06.434146 5007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:06.434168 5007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:06.435247 5007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:06.435279 5007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:06.435297 5007 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:49:06.435310 5007 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:49:06.435326 5007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:06.436725 5007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.0.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:06.436902 5007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:06.437071 5007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:06.441563 5012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:06.441584 5015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:06.441584 5013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:06.441777 5007 server_base.cc:1034] running on GCE node
I20260928 21:49:06.441962 5007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:06.442188 5007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:06.443356 5007 hybrid_clock.cc:648] HybridClock initialized: now 1790632146443337 us; error 28 us; skew 500 ppm
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:06.445685 5007 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:49:06.446107 5007 webserver.cc:460] Webserver started at http://127.3.0.194:37019/ using document root <none> and password file <none>
I20260928 21:49:06.446302 5007 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:06.446355 5007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:06.446473 5007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:06.447507 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "dea6f3803fc94348a8d2ca6aaa4350f8"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "5273952789e2440299963294e37815db"
server_key_iv: "dc7890dfa8601e2c97c2f44548283e8e"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.447829 5007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "dea6f3803fc94348a8d2ca6aaa4350f8"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "5273952789e2440299963294e37815db"
server_key_iv: "dc7890dfa8601e2c97c2f44548283e8e"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.449086 5007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:06.449753 5022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:06.449883 5007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:06.449963 5007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "dea6f3803fc94348a8d2ca6aaa4350f8"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "5273952789e2440299963294e37815db"
server_key_iv: "dc7890dfa8601e2c97c2f44548283e8e"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.450017 5007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:06.459008 5007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:06.460637 5007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:06.460763 5007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:06.461243 5007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:06.461288 5007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:06.461325 5007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:06.461349 5007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:06.467867 5007 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:39313
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:06.467980 5137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:39313 every 8 connection(s)
I20260928 21:49:06.468212 5007 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:49:06.469307 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 5007
I20260928 21:49:06.469398 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:06.469561 3075 external_mini_cluster.cc:1428] Setting key 7859bf0da3c86e28b3bc18bec9523ff1
I20260928 21:49:06.471191 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.461645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:58883 (local address 127.3.0.254:44633)
0928 21:49:06.461708 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.461711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.461934 (+ 223us) server_negotiation.cc:407] Connection header received
0928 21:49:06.462544 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.462545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.462554 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.462634 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:06.463499 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.464194 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.465657 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.465867 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.465935 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.468427 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:06.468428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:06.468428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:06.468435 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:06.469928 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.470336 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.470338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.470338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.470397 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.470605 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.470606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.470606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.470719 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:06.470775 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.471028 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.471105 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:49:06.471341 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.461623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44633 (local address 127.3.0.194:58883)
0928 21:49:06.461794 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.461849 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.462446 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.462764 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.462769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.463127 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:06.463404 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.463410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.464283 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.464284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.465580 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.465583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.465963 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.465965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.466085 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.466925 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:06.466933 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:06.468341 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:06.469992 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.469998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.470001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.470266 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.470446 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.470447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.470447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.470551 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.470771 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:06.470777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:06.470911 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.470998 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.471144 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260928 21:49:06.471793 5138 heartbeater.cc:347] Connected to a master server at 127.3.0.254:44633
I20260928 21:49:06.471881 5138 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:06.472046 5138 heartbeater.cc:511] Master 127.3.0.254:44633 requested a full tablet report, sending...
I20260928 21:49:06.472369 4795 ts_manager.cc:194] Registered new tserver with Master: dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313)
I20260928 21:49:06.472815 4795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:58883
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
I20260928 21:49:06.478202 5145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.472771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44633 (local address 127.3.0.194:34521)
0928 21:49:06.473044 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.473054 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.473131 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.473264 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.473265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.473450 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:06.473513 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.473515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.474127 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.474128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.474918 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.474921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.475216 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.475217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.475287 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.475639 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:06.475644 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:06.476005 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:06.477400 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.477400 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.477401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.477576 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.477711 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.477712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.477712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.477754 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.477965 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:06.477965 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:06.477996 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.478040 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.478125 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":225,"threads_started":1}
I20260928 21:49:06.478225 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.472824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:34521 (local address 127.3.0.254:44633)
0928 21:49:06.472863 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.472864 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.473108 (+ 244us) server_negotiation.cc:407] Connection header received
0928 21:49:06.473193 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.473194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.473205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.473259 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:06.473587 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.474034 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.474989 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.475157 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.475198 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.476080 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:06.476081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:06.476082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:06.476087 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:06.477333 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.477632 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.477633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.477633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.477677 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.477791 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.477792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.477792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.477919 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:06.477967 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.478082 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.478146 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:49:06.490856 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:44633
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:38603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:06.568276 5147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:06.568503 5147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:06.568536 5147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:06.569707 5147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:06.569749 5147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:06.569777 5147 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:49:06.569792 5147 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:49:06.569818 5147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:06.571195 5147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:38603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.0.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:06.571427 5147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:06.571622 5147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:06.576213 5153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:06.576223 5152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:06.576229 5155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:06.576241 5147 server_base.cc:1034] running on GCE node
I20260928 21:49:06.576551 5147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:06.576781 5147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:06.577914 5147 hybrid_clock.cc:648] HybridClock initialized: now 1790632146577901 us; error 30 us; skew 500 ppm
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:06.580087 5147 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:49:06.580574 5147 webserver.cc:460] Webserver started at http://127.3.0.195:41013/ using document root <none> and password file <none>
I20260928 21:49:06.580768 5147 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:06.580811 5147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:06.580935 5147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:06.581905 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "dd0afb0ba688418faf1dcabeac181130"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "a387c052485fa2b63326dbbe3a3e8a0d"
server_key_iv: "b2c40ff939f7e41ec2922cf1bf50f534"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.582224 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "dd0afb0ba688418faf1dcabeac181130"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "a387c052485fa2b63326dbbe3a3e8a0d"
server_key_iv: "b2c40ff939f7e41ec2922cf1bf50f534"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.583360 5147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:49:06.583958 5162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:06.584092 5147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:06.584174 5147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "dd0afb0ba688418faf1dcabeac181130"
format_stamp: "Formatted at 2026-09-28 21:49:06 on dist-test-slave-jrck"
server_key: "a387c052485fa2b63326dbbe3a3e8a0d"
server_key_iv: "b2c40ff939f7e41ec2922cf1bf50f534"
server_key_version: "encryptionkey@0"
I20260928 21:49:06.584246 5147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:06.599635 5147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:06.601385 5147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:06.601498 5147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:06.601999 5147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:06.602039 5147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:06.602061 5147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:06.602072 5147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:06.608417 5147 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:37875
I20260928 21:49:06.608744 5277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:37875 every 8 connection(s)
I20260928 21:49:06.608798 5147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:49:06.610893 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.602356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:33147 (local address 127.3.0.254:44633)
0928 21:49:06.602425 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.602428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.602602 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:49:06.603061 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.603062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.603075 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.603156 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:06.603788 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.604568 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.605595 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.605863 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.605917 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.608486 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:06.608488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:06.608488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:06.608497 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:06.609851 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.610203 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.610203 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.610204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.610250 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.610405 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.610405 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.610406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.610498 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:06.610541 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.610767 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.610821 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:49:06.611088 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.602323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44633 (local address 127.3.0.195:33147)
0928 21:49:06.602486 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.602525 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.602978 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.603219 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.603224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.603514 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:06.603695 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.603700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.604659 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.604660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.605515 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.605517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.605952 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.605953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.606055 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.606540 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:06.606548 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:06.608348 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:06.609945 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.609950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.609951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.610150 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.610285 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.610286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.610286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.610366 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.610540 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:06.610542 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:06.610671 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.610747 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.610898 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260928 21:49:06.611495 5278 heartbeater.cc:347] Connected to a master server at 127.3.0.254:44633
I20260928 21:49:06.611565 5278 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:06.611723 5278 heartbeater.cc:511] Master 127.3.0.254:44633 requested a full tablet report, sending...
I20260928 21:49:06.612407 4794 ts_manager.cc:194] Registered new tserver with Master: dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875)
I20260928 21:49:06.612920 4794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:33147
I20260928 21:49:06.614738 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 5147
I20260928 21:49:06.614842 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:06.615324 3075 external_mini_cluster.cc:1428] Setting key 89adea786275889c190cf1941014a027
I20260928 21:49:06.617336 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:49:06.618362 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.612347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44633 (local address 127.3.0.195:55323)
0928 21:49:06.612562 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.612573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.612629 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.612736 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.612736 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.612916 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:06.612979 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.612980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.613925 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.613926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.614436 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.614437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.614784 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.614785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.614889 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.615271 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:06.615276 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:06.615636 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:06.617468 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.617469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.617469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.617659 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.617870 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:06.617871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:06.617871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:06.617923 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:06.618150 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:06.618150 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:06.618181 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.618223 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.618291 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":129,"threads_started":1}
I20260928 21:49:06.618394 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.612400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:55323 (local address 127.3.0.254:44633)
0928 21:49:06.612444 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.612445 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.612617 (+ 172us) server_negotiation.cc:407] Connection header received
0928 21:49:06.612680 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.612680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.612688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.612732 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:06.613197 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.613838 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.614494 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.614712 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.614773 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.615718 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:06.615719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:06.615719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:06.615726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:06.617380 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.617736 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.617738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.617738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.617802 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.617977 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.617978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.617979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.618095 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:06.618160 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.618247 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.618314 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:49:06 dist-test-slave-jrck krb5kdc[4750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:06.638182 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.632166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.3.0.254:44633)
0928 21:49:06.632220 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.632222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.632230 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:06.632255 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.632256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.632265 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.632315 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:06.632891 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.633370 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.634039 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.634209 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.634250 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.636144 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:06.636146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:06.636147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:06.636153 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:06.637248 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.637560 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.637561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.637561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.637605 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:06.637746 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:06.637746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:06.637746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:06.637838 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:06.637917 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.638044 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.638116 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:49:06.639722 4794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:06.641429 4794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:06.647512 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.643524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.3.0.194:39313)
0928 21:49:06.643738 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.643742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.643753 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:49:06.643860 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.643863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.643880 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.643948 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:06.644127 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.645016 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.646503 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.647113 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.647177 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.647237 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.647380 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.647418 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":105,"threads_started":1}
I20260928 21:49:06.647533 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.643376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:38723 (local address 127.0.0.1:35552)
0928 21:49:06.643683 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.643713 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.643771 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.643973 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.643973 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.643976 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:06.644050 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.644053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.645346 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.645347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.646395 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.646397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.647290 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.647291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.647358 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.647364 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.647416 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.647455 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":65,"threads_started":1}
I20260928 21:49:06.647626 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.643679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.3.0.195:37875)
0928 21:49:06.643897 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.643904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.644117 (+ 213us) server_negotiation.cc:407] Connection header received
0928 21:49:06.644268 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.644270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.644284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.644346 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:06.644545 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.645525 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.646416 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.647135 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.647202 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.647255 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.647496 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.647533 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
I20260928 21:49:06.647742 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.643489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.3.0.193:38723)
0928 21:49:06.643828 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.643831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.643839 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:06.643887 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.643890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.643904 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.643967 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:06.644132 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.645267 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.646452 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.647135 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.647202 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.647254 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.647624 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.647661 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":80,"threads_started":1}
I20260928 21:49:06.647874 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.643583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:37875 (local address 127.0.0.1:33990)
0928 21:49:06.644055 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.644064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.644176 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.644406 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.644407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.644410 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:06.644449 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.644474 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.645592 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.645593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.646288 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.646290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.647290 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.647291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.647358 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.647362 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.647409 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.647457 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":6,"thread_start_us":53,"threads_started":1}
I20260928 21:49:06.648054 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.643361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39313 (local address 127.0.0.1:51784)
0928 21:49:06.643664 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.643704 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.643770 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.643961 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.643965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.643966 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:06.644048 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.644051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.645088 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.645089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.646365 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.646368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.647200 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.647201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.647277 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.647290 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.647356 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.647410 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":26,"thread_start_us":77,"threads_started":1}
I20260928 21:49:06.648379 4932 tablet_service.cc:1467] Processing CreateTablet for tablet d3e50260615242b2ad08a2d976111872 (DEFAULT_TABLE table=test-table [id=8741074f4a774723b92f21d4a67b1570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:06.648777 4932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e50260615242b2ad08a2d976111872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:06.648706 5072 tablet_service.cc:1467] Processing CreateTablet for tablet d3e50260615242b2ad08a2d976111872 (DEFAULT_TABLE table=test-table [id=8741074f4a774723b92f21d4a67b1570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:06.649045 5072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e50260615242b2ad08a2d976111872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:06.648978 5212 tablet_service.cc:1467] Processing CreateTablet for tablet d3e50260615242b2ad08a2d976111872 (DEFAULT_TABLE table=test-table [id=8741074f4a774723b92f21d4a67b1570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:06.649274 5212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e50260615242b2ad08a2d976111872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:06.651943 5298 tablet_bootstrap.cc:492] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: Bootstrap starting.
I20260928 21:49:06.652158 5299 tablet_bootstrap.cc:492] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: Bootstrap starting.
I20260928 21:49:06.652356 5300 tablet_bootstrap.cc:492] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: Bootstrap starting.
I20260928 21:49:06.652604 5298 tablet_bootstrap.cc:654] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:06.652925 5298 log.cc:824] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:06.653045 5299 tablet_bootstrap.cc:654] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:06.653332 5299 log.cc:824] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:06.653635 5298 tablet_bootstrap.cc:492] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: No bootstrap required, opened a new log
I20260928 21:49:06.653702 5298 ts_tablet_manager.cc:1397] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:49:06.654407 5300 tablet_bootstrap.cc:654] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:06.654712 5300 log.cc:824] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:06.655242 5299 tablet_bootstrap.cc:492] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: No bootstrap required, opened a new log
I20260928 21:49:06.655311 5299 ts_tablet_manager.cc:1397] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:49:06.655676 5298 raft_consensus.cc:348] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:06.655848 5298 raft_consensus.cc:374] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:06.655884 5298 raft_consensus.cc:729] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea6f3803fc94348a8d2ca6aaa4350f8, State: Initialized, Role: FOLLOWER
I20260928 21:49:06.655999 5298 consensus_queue.cc:260] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:06.656250 5298 ts_tablet_manager.cc:1428] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:49:06.656272 5138 heartbeater.cc:503] Master 127.3.0.254:44633 was elected leader, sending a full tablet report...
I20260928 21:49:06.656819 5300 tablet_bootstrap.cc:492] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: No bootstrap required, opened a new log
I20260928 21:49:06.656891 5300 ts_tablet_manager.cc:1397] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 21:49:06.657200 5299 raft_consensus.cc:348] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:06.657353 5299 raft_consensus.cc:374] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:06.657393 5299 raft_consensus.cc:729] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3ee5995204477fbf96843b3dfdc271, State: Initialized, Role: FOLLOWER
I20260928 21:49:06.657502 5299 consensus_queue.cc:260] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:06.657745 5299 ts_tablet_manager.cc:1428] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:49:06.657773 4998 heartbeater.cc:503] Master 127.3.0.254:44633 was elected leader, sending a full tablet report...
I20260928 21:49:06.658705 5300 raft_consensus.cc:348] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:06.658820 5300 raft_consensus.cc:374] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:06.658839 5300 raft_consensus.cc:729] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0afb0ba688418faf1dcabeac181130, State: Initialized, Role: FOLLOWER
I20260928 21:49:06.658937 5300 consensus_queue.cc:260] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:06.659157 5300 ts_tablet_manager.cc:1428] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:49:06.659205 5278 heartbeater.cc:503] Master 127.3.0.254:44633 was elected leader, sending a full tablet report...
W20260928 21:49:06.718873 5139 tablet.cc:2369] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:49:06.826726 4999 tablet.cc:2369] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:49:06.859338 5279 tablet.cc:2369] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:49:06.994081 5306 raft_consensus.cc:482] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:06.994202 5306 raft_consensus.cc:504] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:06.994545 5306 leader_election.cc:288] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723), dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313)
I20260928 21:49:06.997238 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.994772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:48247 (local address 127.3.0.193:38723)
0928 21:49:06.994815 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.994817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.994954 (+ 137us) server_negotiation.cc:407] Connection header received
0928 21:49:06.995017 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.995018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.995031 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.995085 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:06.995219 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.995660 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.996576 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.996936 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.996997 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.997007 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.997135 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.997168 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:49:06.997251 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.994714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39313 (local address 127.3.0.195:36783)
0928 21:49:06.994776 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.994788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.994826 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.994984 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.994984 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.994986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:06.995054 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.995056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.995686 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.995688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.996493 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.996495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.997000 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.997001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.997071 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.997074 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.997119 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.997158 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":40}
I20260928 21:49:06.997246 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.994767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:36783 (local address 127.3.0.194:39313)
0928 21:49:06.994815 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:06.994817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:06.994826 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:49:06.994881 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:06.994882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:06.994891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:06.994954 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:06.995118 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.995615 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.996579 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.996931 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.996988 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.996998 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:06.997135 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 21:49:06.997170 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:49:06.997275 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:06.994714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:38723 (local address 127.3.0.195:48247)
0928 21:49:06.994888 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:06.994898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:06.994952 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:06.995092 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:06.995092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:06.995094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:06.995157 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.995159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.995744 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.995745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.996500 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:06.996502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:06.997007 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:06.997008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:06.997071 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:06.997074 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:06.997119 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:49:06.997159 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":12,"thread_start_us":87,"threads_started":1}
I20260928 21:49:06.997578 4950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e50260615242b2ad08a2d976111872" candidate_uuid: "dd0afb0ba688418faf1dcabeac181130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3ee5995204477fbf96843b3dfdc271" is_pre_election: true
I20260928 21:49:06.997586 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e50260615242b2ad08a2d976111872" candidate_uuid: "dd0afb0ba688418faf1dcabeac181130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" is_pre_election: true
I20260928 21:49:06.997728 5092 raft_consensus.cc:2454] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd0afb0ba688418faf1dcabeac181130 in term 0.
I20260928 21:49:06.997736 4950 raft_consensus.cc:2454] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd0afb0ba688418faf1dcabeac181130 in term 0.
I20260928 21:49:06.997944 5164 leader_election.cc:302] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3ee5995204477fbf96843b3dfdc271, dd0afb0ba688418faf1dcabeac181130; no voters:
I20260928 21:49:06.998070 5306 raft_consensus.cc:2781] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:06.998124 5306 raft_consensus.cc:482] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:06.998140 5306 raft_consensus.cc:3037] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:06.999032 5306 raft_consensus.cc:504] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:06.999135 5306 leader_election.cc:288] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723), dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313)
I20260928 21:49:06.999270 4950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e50260615242b2ad08a2d976111872" candidate_uuid: "dd0afb0ba688418faf1dcabeac181130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3ee5995204477fbf96843b3dfdc271"
I20260928 21:49:06.999300 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e50260615242b2ad08a2d976111872" candidate_uuid: "dd0afb0ba688418faf1dcabeac181130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8"
I20260928 21:49:06.999346 4950 raft_consensus.cc:3037] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:06.999363 5092 raft_consensus.cc:3037] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:06.999976 4950 raft_consensus.cc:2454] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd0afb0ba688418faf1dcabeac181130 in term 1.
I20260928 21:49:06.999984 5092 raft_consensus.cc:2454] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd0afb0ba688418faf1dcabeac181130 in term 1.
I20260928 21:49:07.000140 5164 leader_election.cc:302] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3ee5995204477fbf96843b3dfdc271, dd0afb0ba688418faf1dcabeac181130; no voters:
I20260928 21:49:07.000260 5306 raft_consensus.cc:2781] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:07.000476 5306 raft_consensus.cc:686] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 1 LEADER]: Becoming Leader. State: Replica: dd0afb0ba688418faf1dcabeac181130, State: Running, Role: LEADER
I20260928 21:49:07.000571 5306 consensus_queue.cc:237] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.001310 4794 catalog_manager.cc:5495] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd0afb0ba688418faf1dcabeac181130 (127.3.0.195). New cstate: current_term: 1 leader_uuid: "dd0afb0ba688418faf1dcabeac181130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } health_report { overall_health: HEALTHY } } }
Sep 28 21:49:07 dist-test-slave-jrck krb5kdc[4750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632146, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:07.107232 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.101116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.3.0.254:44633)
0928 21:49:07.101189 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.101192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.101258 (+ 66us) server_negotiation.cc:407] Connection header received
0928 21:49:07.101311 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.101312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.101320 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.101368 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:07.101781 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.102255 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.102888 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.103057 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.103099 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.104939 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:07.104940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:07.104940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:07.104947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:07.106214 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:07.106547 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:07.106548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:07.106549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:07.106594 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:07.106747 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:07.106748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:07.106748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:07.106849 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:07.106909 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.107044 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.107133 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260928 21:49:07.107306 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.101064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44633 (local address 127.0.0.1:48988)
0928 21:49:07.101193 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:07.101202 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:07.101263 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:07.101381 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:07.101381 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:07.101676 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:07.101727 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.101729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.102308 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.102309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.102824 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.102825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.103102 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.103102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.103137 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.103569 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:07.103574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:07.104861 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:07.106285 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:07.106293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:07.106295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:07.106485 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:07.106642 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:07.106643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:07.106643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:07.106698 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:07.106901 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:07.106908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:07.106956 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 21:49:07.107017 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:49:07.107198 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":72,"thread_start_us":65,"threads_started":1}
I20260928 21:49:07.109072 4794 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:48988:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:49:07.109249 4794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:07.110607 4932 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfe8517aef54e519ed001ecde419772 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10fe3ec06d6a4e34982351eb34555850]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:49:07.110605 5072 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfe8517aef54e519ed001ecde419772 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10fe3ec06d6a4e34982351eb34555850]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:49:07.110648 5212 tablet_service.cc:1467] Processing CreateTablet for tablet 1cfe8517aef54e519ed001ecde419772 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=10fe3ec06d6a4e34982351eb34555850]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:49:07.110762 4932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cfe8517aef54e519ed001ecde419772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:07.110764 5072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cfe8517aef54e519ed001ecde419772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:07.110773 5212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cfe8517aef54e519ed001ecde419772. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:07.111971 5299 tablet_bootstrap.cc:492] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: Bootstrap starting.
I20260928 21:49:07.112205 5300 tablet_bootstrap.cc:492] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: Bootstrap starting.
I20260928 21:49:07.112205 5298 tablet_bootstrap.cc:492] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: Bootstrap starting.
I20260928 21:49:07.112362 5299 tablet_bootstrap.cc:654] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:07.112607 5300 tablet_bootstrap.cc:654] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:07.112607 5298 tablet_bootstrap.cc:654] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:07.113158 5299 tablet_bootstrap.cc:492] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: No bootstrap required, opened a new log
I20260928 21:49:07.113166 5298 tablet_bootstrap.cc:492] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: No bootstrap required, opened a new log
I20260928 21:49:07.113201 5299 ts_tablet_manager.cc:1397] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:07.113209 5298 ts_tablet_manager.cc:1397] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:07.113330 5300 tablet_bootstrap.cc:492] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: No bootstrap required, opened a new log
I20260928 21:49:07.113340 5298 raft_consensus.cc:348] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.113366 5300 ts_tablet_manager.cc:1397] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:07.113386 5298 raft_consensus.cc:374] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:07.113399 5298 raft_consensus.cc:729] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea6f3803fc94348a8d2ca6aaa4350f8, State: Initialized, Role: FOLLOWER
I20260928 21:49:07.113353 5299 raft_consensus.cc:348] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.113416 5299 raft_consensus.cc:374] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:07.113437 5299 raft_consensus.cc:729] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3ee5995204477fbf96843b3dfdc271, State: Initialized, Role: FOLLOWER
I20260928 21:49:07.113433 5298 consensus_queue.cc:260] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.113482 5299 consensus_queue.cc:260] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.113528 5298 ts_tablet_manager.cc:1428] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:07.113499 5300 raft_consensus.cc:348] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.113554 5300 raft_consensus.cc:374] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:07.113574 5300 raft_consensus.cc:729] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0afb0ba688418faf1dcabeac181130, State: Initialized, Role: FOLLOWER
I20260928 21:49:07.113571 5299 ts_tablet_manager.cc:1428] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:07.113611 5300 consensus_queue.cc:260] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.113626 5304 tablet_replica.cc:439] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } } }
I20260928 21:49:07.113624 5305 tablet_replica.cc:439] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } } }
I20260928 21:49:07.113687 5304 tablet_replica.cc:442] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:07.113693 5305 tablet_replica.cc:442] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:07.113703 5300 ts_tablet_manager.cc:1428] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:07.113792 5306 tablet_replica.cc:439] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } } }
I20260928 21:49:07.113838 5306 tablet_replica.cc:442] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:07.214902 5304 raft_consensus.cc:482] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:07.215000 5304 raft_consensus.cc:504] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.215353 5304 leader_election.cc:288] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723), dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875)
I20260928 21:49:07.219789 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.215653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:46711 (local address 127.3.0.193:38723)
0928 21:49:07.215701 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.215703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.215763 (+ 60us) server_negotiation.cc:407] Connection header received
0928 21:49:07.215836 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.215837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.215845 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.215909 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:07.216085 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.216859 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.218860 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.219416 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.219488 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.219500 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.219675 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.219712 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:49:07.219797 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.215590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:43221 (local address 127.3.0.195:37875)
0928 21:49:07.215718 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.215719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.215752 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:49:07.215831 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.215832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.215844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.215912 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:07.216084 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.216854 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.218860 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.219451 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.219509 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.219519 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.219675 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.219714 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260928 21:49:07.219820 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.215529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:38723 (local address 127.3.0.194:46711)
0928 21:49:07.215676 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:07.215689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:07.215768 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:07.215923 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:07.215924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:07.215927 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:07.216021 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.216024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.216923 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.216924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.218765 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.218769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.219523 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.219525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.219602 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.219608 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:49:07.219663 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:49:07.219710 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":120,"thread_start_us":58,"threads_started":1}
I20260928 21:49:07.219821 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.215529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:37875 (local address 127.3.0.194:43221)
0928 21:49:07.215701 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:07.215710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:07.215768 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:07.215923 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:07.215924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:07.215926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:07.216025 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.216026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.216923 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.216924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.218765 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.218769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.219532 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.219532 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.219602 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.219607 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:49:07.219663 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:49:07.219710 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":552,"thread_start_us":59,"threads_started":1}
I20260928 21:49:07.220094 4950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfe8517aef54e519ed001ecde419772" candidate_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3ee5995204477fbf96843b3dfdc271" is_pre_election: true
I20260928 21:49:07.220199 4950 raft_consensus.cc:2454] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea6f3803fc94348a8d2ca6aaa4350f8 in term 0.
I20260928 21:49:07.220247 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfe8517aef54e519ed001ecde419772" candidate_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0afb0ba688418faf1dcabeac181130" is_pre_election: true
I20260928 21:49:07.220371 5232 raft_consensus.cc:2454] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea6f3803fc94348a8d2ca6aaa4350f8 in term 0.
I20260928 21:49:07.220378 5024 leader_election.cc:302] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3ee5995204477fbf96843b3dfdc271, dea6f3803fc94348a8d2ca6aaa4350f8; no voters:
I20260928 21:49:07.220531 5304 raft_consensus.cc:2781] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:07.220580 5304 raft_consensus.cc:482] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:07.220599 5304 raft_consensus.cc:3037] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:07.221223 5304 raft_consensus.cc:504] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.221338 5304 leader_election.cc:288] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723), dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875)
I20260928 21:49:07.221484 4950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfe8517aef54e519ed001ecde419772" candidate_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3ee5995204477fbf96843b3dfdc271"
I20260928 21:49:07.221489 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cfe8517aef54e519ed001ecde419772" candidate_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0afb0ba688418faf1dcabeac181130"
I20260928 21:49:07.221552 4950 raft_consensus.cc:3037] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:07.221554 5232 raft_consensus.cc:3037] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:07.222136 4950 raft_consensus.cc:2454] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea6f3803fc94348a8d2ca6aaa4350f8 in term 1.
I20260928 21:49:07.222138 5232 raft_consensus.cc:2454] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea6f3803fc94348a8d2ca6aaa4350f8 in term 1.
I20260928 21:49:07.222282 5026 leader_election.cc:302] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd0afb0ba688418faf1dcabeac181130, dea6f3803fc94348a8d2ca6aaa4350f8; no voters:
I20260928 21:49:07.222427 5304 raft_consensus.cc:2781] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:07.222534 5304 raft_consensus.cc:686] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 LEADER]: Becoming Leader. State: Replica: dea6f3803fc94348a8d2ca6aaa4350f8, State: Running, Role: LEADER
I20260928 21:49:07.222623 5304 consensus_queue.cc:237] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.223023 5322 tablet_replica.cc:439] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: TxnStatusTablet state changed. Reason: New leader dea6f3803fc94348a8d2ca6aaa4350f8. Latest consensus state: current_term: 1 leader_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } } }
I20260928 21:49:07.223109 5322 tablet_replica.cc:442] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:49:07.223345 4792 catalog_manager.cc:5495] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194). New cstate: current_term: 1 leader_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } health_report { overall_health: UNKNOWN } } }
I20260928 21:49:07.223639 5324 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:49:07.223760 5324 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:49:07.229323 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.226527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.3.0.194:39313)
0928 21:49:07.226562 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.226564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.226606 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:49:07.226693 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.226694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.226707 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.226759 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:07.226881 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.227349 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.228482 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.228693 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.228775 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.228970 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:07.229080 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:07.229148 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.229194 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.229219 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:49:07.229323 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.226486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39313 (local address 127.0.0.1:51798)
0928 21:49:07.226563 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:07.226574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:07.226623 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:07.226765 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:07.226766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:07.226768 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:07.226824 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.226827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.227416 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.227417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.228337 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.228340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.228804 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.228805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.228872 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.228888 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:07.229139 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:07.229148 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 21:49:07.229192 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:49:07.229244 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 21:49:07.231014 4950 raft_consensus.cc:1261] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Refusing update from remote peer dea6f3803fc94348a8d2ca6aaa4350f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:07.231328 5232 raft_consensus.cc:1261] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Refusing update from remote peer dea6f3803fc94348a8d2ca6aaa4350f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:07.231294 5304 consensus_queue.cc:1035] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:07.231559 5304 consensus_queue.cc:1035] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:07.232512 5305 tablet_replica.cc:439] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: TxnStatusTablet state changed. Reason: New leader dea6f3803fc94348a8d2ca6aaa4350f8. Latest consensus state: current_term: 1 leader_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } } }
I20260928 21:49:07.232595 5305 tablet_replica.cc:442] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:07.233528 5306 tablet_replica.cc:439] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: TxnStatusTablet state changed. Reason: New leader dea6f3803fc94348a8d2ca6aaa4350f8. Latest consensus state: current_term: 1 leader_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } } }
I20260928 21:49:07.233615 5306 tablet_replica.cc:442] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:07.234162 5304 tablet_replica.cc:439] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } } }
I20260928 21:49:07.234239 5304 tablet_replica.cc:442] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:49:07.234206 5322 tablet_replica.cc:439] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } } }
I20260928 21:49:07.234283 5322 tablet_replica.cc:442] T 1cfe8517aef54e519ed001ecde419772 P dea6f3803fc94348a8d2ca6aaa4350f8: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:49:07.234346 5327 mvcc.cc:205] Tried to move back new op lower bound from 7334429275055824896 to 7334429275024355328. Current Snapshot: MvccSnapshot[applied={T|T < 7334429275055824896}]
I20260928 21:49:07.234536 5306 tablet_replica.cc:439] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } } }
I20260928 21:49:07.234609 5306 tablet_replica.cc:442] T 1cfe8517aef54e519ed001ecde419772 P dd0afb0ba688418faf1dcabeac181130: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:07.234754 5305 tablet_replica.cc:439] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } } }
I20260928 21:49:07.234846 5305 tablet_replica.cc:442] T 1cfe8517aef54e519ed001ecde419772 P 3a3ee5995204477fbf96843b3dfdc271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:07.235260 5330 mvcc.cc:205] Tried to move back new op lower bound from 7334429275055824896 to 7334429275024355328. Current Snapshot: MvccSnapshot[applied={T|T < 7334429275055824896}]
I20260928 21:49:07.239512 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.236884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.3.0.195:37875)
0928 21:49:07.236925 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.236927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.236989 (+ 62us) server_negotiation.cc:407] Connection header received
0928 21:49:07.237036 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.237037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.237055 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.237131 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:07.237220 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.237959 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.238711 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.239022 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.239084 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.239214 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:07.239327 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:07.239402 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.239419 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.239441 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:49:07.244267 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.241711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39313 (local address 127.3.0.195:57633)
0928 21:49:07.241885 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:07.241895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:07.241945 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:07.242075 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:07.242075 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:07.242077 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:07.242123 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.242124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.242966 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.242967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.243521 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.243522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.243941 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.243942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.243983 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.243995 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:07.244151 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:07.244152 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:49:07.244178 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 21:49:07.244208 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260928 21:49:07.244270 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.241784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:57633 (local address 127.3.0.194:39313)
0928 21:49:07.241822 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.241824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.241948 (+ 124us) server_negotiation.cc:407] Connection header received
0928 21:49:07.241990 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.241990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.242001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.242054 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:07.242170 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.242902 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.243584 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.243882 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.243945 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.244069 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:07.244100 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:07.244142 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.244183 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.244200 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:49:07.246356 4950 raft_consensus.cc:1261] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Refusing update from remote peer dd0afb0ba688418faf1dcabeac181130: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:07.246387 5092 raft_consensus.cc:1261] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Refusing update from remote peer dd0afb0ba688418faf1dcabeac181130: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:07.246613 5308 consensus_queue.cc:1035] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:07.246748 5306 consensus_queue.cc:1035] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:07.248857 5328 mvcc.cc:205] Tried to move back new op lower bound from 7334429275119403008 to 7334429274114793472. Current Snapshot: MvccSnapshot[applied={T|T < 7334429275119403008}]
I20260928 21:49:07.254482 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.252112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:50187 (local address 127.3.0.195:37875)
0928 21:49:07.252157 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.252160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.252189 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:49:07.252244 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.252245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.252254 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.252303 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:07.252426 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.252913 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.253774 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.254037 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.254073 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.254259 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:07.254291 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:07.254334 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.254398 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.254415 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:49:07.254518 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.251988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:37875 (local address 127.3.0.194:50187)
0928 21:49:07.252114 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:07.252146 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:07.252200 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:07.252306 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:07.252307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:07.252308 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:07.252362 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.252363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.252971 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.252972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.253686 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.253688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.254117 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.254117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.254171 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.254183 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:07.254339 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:07.254341 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:49:07.254393 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:49:07.254433 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260928 21:49:07.261706 4794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:07.261809 4794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:07.262620 4794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8: Sending DeleteTablet for 3 replicas of tablet d3e50260615242b2ad08a2d976111872
I20260928 21:49:07.262852 5072 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e50260615242b2ad08a2d976111872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:07 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:51784
I20260928 21:49:07.263065 4794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:07.263195 4794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:07.263200 5212 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e50260615242b2ad08a2d976111872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:07 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:33990
I20260928 21:49:07.263449 5072 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e50260615242b2ad08a2d976111872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:07 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:51784
I20260928 21:49:07.263551 4932 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e50260615242b2ad08a2d976111872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:07 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:35552
I20260928 21:49:07.264398 4932 tablet_service.cc:1467] Processing CreateTablet for tablet df2492a3d98b487f9985576e3fced62b (DEFAULT_TABLE table=test-table [id=c76304518a9c499da9c6e94ac93ebb83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:07.264551 5072 tablet_service.cc:1467] Processing CreateTablet for tablet df2492a3d98b487f9985576e3fced62b (DEFAULT_TABLE table=test-table [id=c76304518a9c499da9c6e94ac93ebb83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:07.264576 4932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df2492a3d98b487f9985576e3fced62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:07.264667 5072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df2492a3d98b487f9985576e3fced62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:07.265589 5298 tablet_bootstrap.cc:492] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8: Bootstrap starting.
I20260928 21:49:07.265645 5212 tablet_service.cc:1467] Processing CreateTablet for tablet df2492a3d98b487f9985576e3fced62b (DEFAULT_TABLE table=test-table [id=c76304518a9c499da9c6e94ac93ebb83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:07.265743 5212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df2492a3d98b487f9985576e3fced62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:07.265858 5299 tablet_bootstrap.cc:492] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271: Bootstrap starting.
I20260928 21:49:07.265952 5298 tablet_bootstrap.cc:654] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:07.266218 5299 tablet_bootstrap.cc:654] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:07.266942 5300 tablet_bootstrap.cc:492] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130: Bootstrap starting.
I20260928 21:49:07.267257 5348 tablet_replica.cc:330] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: stopping tablet replica
I20260928 21:49:07.267319 5300 tablet_bootstrap.cc:654] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:07.267354 5348 raft_consensus.cc:2229] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:07.267484 5348 raft_consensus.cc:2258] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:07.267630 5349 tablet_replica.cc:330] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: stopping tablet replica
I20260928 21:49:07.267731 5349 raft_consensus.cc:2229] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:07.267819 5349 raft_consensus.cc:2258] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:07.267932 5348 ts_tablet_manager.cc:1905] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:07.268117 5349 ts_tablet_manager.cc:1905] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:07.269907 5349 ts_tablet_manager.cc:1918] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:07.270035 5349 log.cc:1196] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/d3e50260615242b2ad08a2d976111872
I20260928 21:49:07.270093 5348 ts_tablet_manager.cc:1918] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:07.270153 5348 log.cc:1196] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/d3e50260615242b2ad08a2d976111872
I20260928 21:49:07.270179 5299 tablet_bootstrap.cc:492] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271: No bootstrap required, opened a new log
I20260928 21:49:07.270231 5299 ts_tablet_manager.cc:1397] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 21:49:07.270364 5299 raft_consensus.cc:348] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.270411 5348 ts_tablet_manager.cc:1939] T d3e50260615242b2ad08a2d976111872 P dd0afb0ba688418faf1dcabeac181130: Deleting consensus metadata
I20260928 21:49:07.270413 5347 tablet_replica.cc:330] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: stopping tablet replica
I20260928 21:49:07.270438 5299 raft_consensus.cc:374] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:07.270457 5299 raft_consensus.cc:729] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3ee5995204477fbf96843b3dfdc271, State: Initialized, Role: FOLLOWER
I20260928 21:49:07.270494 5299 consensus_queue.cc:260] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.270510 5347 raft_consensus.cc:2229] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:07.270522 5349 ts_tablet_manager.cc:1939] T d3e50260615242b2ad08a2d976111872 P 3a3ee5995204477fbf96843b3dfdc271: Deleting consensus metadata
I20260928 21:49:07.270567 5347 raft_consensus.cc:2258] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:07.270569 5299 ts_tablet_manager.cc:1428] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:07.270752 4782 catalog_manager.cc:4841] TS dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875): tablet d3e50260615242b2ad08a2d976111872 (table test-table [id=8741074f4a774723b92f21d4a67b1570]) successfully deleted
I20260928 21:49:07.270895 5347 ts_tablet_manager.cc:1905] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:07.270967 4780 catalog_manager.cc:4841] TS 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723): tablet d3e50260615242b2ad08a2d976111872 (table test-table [id=8741074f4a774723b92f21d4a67b1570]) successfully deleted
I20260928 21:49:07.271041 5300 tablet_bootstrap.cc:492] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130: No bootstrap required, opened a new log
I20260928 21:49:07.271101 5300 ts_tablet_manager.cc:1397] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 21:49:07.271239 5300 raft_consensus.cc:348] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.271291 5300 raft_consensus.cc:374] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:07.271306 5300 raft_consensus.cc:729] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0afb0ba688418faf1dcabeac181130, State: Initialized, Role: FOLLOWER
I20260928 21:49:07.271346 5300 consensus_queue.cc:260] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.271436 5300 ts_tablet_manager.cc:1428] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:07.272486 5347 ts_tablet_manager.cc:1918] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:07.272557 5347 log.cc:1196] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/d3e50260615242b2ad08a2d976111872
I20260928 21:49:07.272845 5347 ts_tablet_manager.cc:1939] T d3e50260615242b2ad08a2d976111872 P dea6f3803fc94348a8d2ca6aaa4350f8: Deleting consensus metadata
I20260928 21:49:07.273227 4782 catalog_manager.cc:4841] TS dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313): tablet d3e50260615242b2ad08a2d976111872 (table test-table [id=8741074f4a774723b92f21d4a67b1570]) successfully deleted
W20260928 21:49:07.273348 4782 catalog_manager.cc:4808] TS dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313): delete failed for tablet d3e50260615242b2ad08a2d976111872 because the tablet was not found. No further retry: Not found: Tablet not found: d3e50260615242b2ad08a2d976111872
I20260928 21:49:07.273622 5298 tablet_bootstrap.cc:492] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8: No bootstrap required, opened a new log
I20260928 21:49:07.273672 5298 ts_tablet_manager.cc:1397] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 21:49:07.273792 5298 raft_consensus.cc:348] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.273859 5298 raft_consensus.cc:374] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:07.273885 5298 raft_consensus.cc:729] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea6f3803fc94348a8d2ca6aaa4350f8, State: Initialized, Role: FOLLOWER
I20260928 21:49:07.273926 5298 consensus_queue.cc:260] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.274000 5298 ts_tablet_manager.cc:1428] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:07.587548 5305 raft_consensus.cc:482] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:07.587651 5305 raft_consensus.cc:504] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.587975 5305 leader_election.cc:288] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313), dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875)
I20260928 21:49:07.590498 5304 raft_consensus.cc:482] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:07.590579 5304 raft_consensus.cc:504] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.590715 5304 leader_election.cc:288] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723), dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875)
I20260928 21:49:07.590890 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.588217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:40247 (local address 127.3.0.194:39313)
0928 21:49:07.588282 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.588284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.588359 (+ 75us) server_negotiation.cc:407] Connection header received
0928 21:49:07.588500 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.588501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.588512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.588566 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:07.588705 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.589203 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.590163 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.590544 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.590602 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.590613 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.590780 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.590816 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:49:07.590854 4950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2492a3d98b487f9985576e3fced62b" candidate_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3ee5995204477fbf96843b3dfdc271" is_pre_election: true
I20260928 21:49:07.590927 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.588389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:35191 (local address 127.3.0.195:37875)
0928 21:49:07.588430 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.588432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.588504 (+ 72us) server_negotiation.cc:407] Connection header received
0928 21:49:07.588569 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.588570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.588579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.588632 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:07.588758 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.589195 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.590162 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.590572 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.590635 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.590646 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.590836 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.590871 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:07.590937 4950 raft_consensus.cc:2454] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea6f3803fc94348a8d2ca6aaa4350f8 in term 0.
I20260928 21:49:07.591022 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2492a3d98b487f9985576e3fced62b" candidate_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0afb0ba688418faf1dcabeac181130" is_pre_election: true
I20260928 21:49:07.591066 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.588147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39313 (local address 127.3.0.193:40247)
0928 21:49:07.588274 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:07.588285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:07.588371 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:07.588578 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:07.588578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:07.588581 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:07.588644 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.588646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.589300 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.589302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.590080 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.590082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.590645 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.590646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.590701 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.590706 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:49:07.590956 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 21:49:07.590994 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":24,"thread_start_us":60,"threads_started":1}
I20260928 21:49:07.591094 5232 raft_consensus.cc:2454] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea6f3803fc94348a8d2ca6aaa4350f8 in term 0.
I20260928 21:49:07.591130 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.588311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:37875 (local address 127.3.0.193:35191)
0928 21:49:07.588433 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:07.588442 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:07.588517 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:07.588644 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:07.588645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:07.588647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:07.588699 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.588701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.589275 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.589276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.590077 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.590079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.590678 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.590679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.590750 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.590753 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:07.591000 (+ 247us) client_negotiation.cc:240] Negotiation successful
0928 21:49:07.591043 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260928 21:49:07.591233 5026 leader_election.cc:302] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a3ee5995204477fbf96843b3dfdc271, dd0afb0ba688418faf1dcabeac181130, dea6f3803fc94348a8d2ca6aaa4350f8; no voters:
I20260928 21:49:07.591388 5304 raft_consensus.cc:2781] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:07.591418 5304 raft_consensus.cc:482] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:07.591439 5304 raft_consensus.cc:3037] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:07.591419 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2492a3d98b487f9985576e3fced62b" candidate_uuid: "3a3ee5995204477fbf96843b3dfdc271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" is_pre_election: true
I20260928 21:49:07.591794 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2492a3d98b487f9985576e3fced62b" candidate_uuid: "3a3ee5995204477fbf96843b3dfdc271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0afb0ba688418faf1dcabeac181130" is_pre_election: true
I20260928 21:49:07.591861 5232 raft_consensus.cc:2454] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3ee5995204477fbf96843b3dfdc271 in term 0.
I20260928 21:49:07.592001 4886 leader_election.cc:302] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3ee5995204477fbf96843b3dfdc271, dd0afb0ba688418faf1dcabeac181130; no voters:
I20260928 21:49:07.592140 5305 raft_consensus.cc:2781] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:07.592186 5305 raft_consensus.cc:482] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:07.592207 5305 raft_consensus.cc:3037] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:07.592202 5304 raft_consensus.cc:504] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.592319 5304 leader_election.cc:288] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723), dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875)
I20260928 21:49:07.592321 5092 raft_consensus.cc:2379] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a3ee5995204477fbf96843b3dfdc271 in current term 1: Already voted for candidate dea6f3803fc94348a8d2ca6aaa4350f8 in this term.
I20260928 21:49:07.592466 4950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2492a3d98b487f9985576e3fced62b" candidate_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3ee5995204477fbf96843b3dfdc271"
I20260928 21:49:07.592512 5232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2492a3d98b487f9985576e3fced62b" candidate_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0afb0ba688418faf1dcabeac181130"
I20260928 21:49:07.592568 5232 raft_consensus.cc:3037] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:07.593052 5305 raft_consensus.cc:504] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.593186 4950 raft_consensus.cc:2379] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dea6f3803fc94348a8d2ca6aaa4350f8 in current term 1: Already voted for candidate 3a3ee5995204477fbf96843b3dfdc271 in this term.
I20260928 21:49:07.593185 5305 leader_election.cc:288] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [CANDIDATE]: Term 1 election: Requested vote from peers dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313), dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875)
I20260928 21:49:07.593286 5232 raft_consensus.cc:2454] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea6f3803fc94348a8d2ca6aaa4350f8 in term 1.
I20260928 21:49:07.593329 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2492a3d98b487f9985576e3fced62b" candidate_uuid: "3a3ee5995204477fbf96843b3dfdc271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0afb0ba688418faf1dcabeac181130"
I20260928 21:49:07.593398 5026 leader_election.cc:302] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd0afb0ba688418faf1dcabeac181130, dea6f3803fc94348a8d2ca6aaa4350f8; no voters:
I20260928 21:49:07.593429 5231 raft_consensus.cc:2379] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3a3ee5995204477fbf96843b3dfdc271 in current term 1: Already voted for candidate dea6f3803fc94348a8d2ca6aaa4350f8 in this term.
I20260928 21:49:07.593586 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df2492a3d98b487f9985576e3fced62b" candidate_uuid: "3a3ee5995204477fbf96843b3dfdc271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8"
I20260928 21:49:07.593586 5304 raft_consensus.cc:2781] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:07.593686 5304 raft_consensus.cc:686] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 LEADER]: Becoming Leader. State: Replica: dea6f3803fc94348a8d2ca6aaa4350f8, State: Running, Role: LEADER
I20260928 21:49:07.593736 5304 consensus_queue.cc:237] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:07.594112 4886 leader_election.cc:302] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a3ee5995204477fbf96843b3dfdc271; no voters: dd0afb0ba688418faf1dcabeac181130, dea6f3803fc94348a8d2ca6aaa4350f8
I20260928 21:49:07.594185 5305 raft_consensus.cc:2726] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 21:49:07.594354 4794 catalog_manager.cc:5495] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194). New cstate: current_term: 1 leader_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } health_report { overall_health: UNKNOWN } } }
I20260928 21:49:07.624163 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.622338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.3.0.194:39313)
0928 21:49:07.622397 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.622399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.622407 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:07.622428 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.622428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.622436 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.622476 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:07.622558 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.623017 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.623673 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.623850 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.623895 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.623986 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:07.624011 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:07.624041 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.624073 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.624087 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:49:07.627856 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.626130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:51923 (local address 127.3.0.194:39313)
0928 21:49:07.626165 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:07.626167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:07.626173 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:49:07.626210 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:07.626211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:07.626217 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:07.626252 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:07.626341 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.626769 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.627390 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.627551 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.627591 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.627665 (+ 74us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:07.627701 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:07.627730 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:07.627762 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 21:49:07.627790 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":19}
I20260928 21:49:07.627869 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:07.626067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39313 (local address 127.3.0.194:51923)
0928 21:49:07.626119 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:07.626130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:07.626165 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:07.626256 (+ 91us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:07.626256 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:07.626257 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:07.626290 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.626291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.626815 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.626816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.627337 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:07.627338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:07.627595 (+ 257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:07.627595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:07.627627 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:07.627632 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:07.627731 (+ 99us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:07.627732 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:49:07.627759 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 21:49:07.627803 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":4}
I20260928 21:49:07.629657 5231 raft_consensus.cc:1261] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Refusing update from remote peer dea6f3803fc94348a8d2ca6aaa4350f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:07.629709 4950 raft_consensus.cc:1261] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Refusing update from remote peer dea6f3803fc94348a8d2ca6aaa4350f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:07.629849 5333 consensus_queue.cc:1035] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:07.630007 5333 consensus_queue.cc:1035] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:49:07.633948 5327 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:49:09.150027 4794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:09.150125 4794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:09.150841 4794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8: Sending DeleteTablet for 3 replicas of tablet df2492a3d98b487f9985576e3fced62b
I20260928 21:49:09.151073 4932 tablet_service.cc:1514] Processing DeleteTablet for tablet df2492a3d98b487f9985576e3fced62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:09 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:35552
I20260928 21:49:09.151089 5072 tablet_service.cc:1514] Processing DeleteTablet for tablet df2492a3d98b487f9985576e3fced62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:09 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:51784
I20260928 21:49:09.151114 5212 tablet_service.cc:1514] Processing DeleteTablet for tablet df2492a3d98b487f9985576e3fced62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:09 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:33990
I20260928 21:49:09.151268 5381 tablet_replica.cc:330] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130: stopping tablet replica
I20260928 21:49:09.151322 5381 raft_consensus.cc:2229] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:09.151424 5380 tablet_replica.cc:330] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8: stopping tablet replica
I20260928 21:49:09.151477 5380 raft_consensus.cc:2229] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:09.151549 5380 raft_consensus.cc:2258] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:09.151613 5381 raft_consensus.cc:2258] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:09.151821 5380 ts_tablet_manager.cc:1905] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:09.153376 5380 ts_tablet_manager.cc:1918] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:49:09.153455 5380 log.cc:1196] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/df2492a3d98b487f9985576e3fced62b
I20260928 21:49:09.153661 5379 tablet_replica.cc:330] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271: stopping tablet replica
I20260928 21:49:09.153674 5380 ts_tablet_manager.cc:1939] T df2492a3d98b487f9985576e3fced62b P dea6f3803fc94348a8d2ca6aaa4350f8: Deleting consensus metadata
I20260928 21:49:09.153718 5379 raft_consensus.cc:2229] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:09.153772 5379 raft_consensus.cc:2258] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:09.153879 5381 ts_tablet_manager.cc:1905] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:09.153978 5379 ts_tablet_manager.cc:1905] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:09.155007 5379 ts_tablet_manager.cc:1918] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:49:09.155056 5379 log.cc:1196] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/df2492a3d98b487f9985576e3fced62b
I20260928 21:49:09.155234 5379 ts_tablet_manager.cc:1939] T df2492a3d98b487f9985576e3fced62b P 3a3ee5995204477fbf96843b3dfdc271: Deleting consensus metadata
I20260928 21:49:09.155337 5381 ts_tablet_manager.cc:1918] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:49:09.155397 5381 log.cc:1196] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/df2492a3d98b487f9985576e3fced62b
I20260928 21:49:09.155575 5381 ts_tablet_manager.cc:1939] T df2492a3d98b487f9985576e3fced62b P dd0afb0ba688418faf1dcabeac181130: Deleting consensus metadata
I20260928 21:49:09.155660 4780 catalog_manager.cc:4841] TS 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723): tablet df2492a3d98b487f9985576e3fced62b (table test-table [id=c76304518a9c499da9c6e94ac93ebb83]) successfully deleted
I20260928 21:49:09.155838 4782 catalog_manager.cc:4841] TS dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875): tablet df2492a3d98b487f9985576e3fced62b (table test-table [id=c76304518a9c499da9c6e94ac93ebb83]) successfully deleted
I20260928 21:49:09.156708 4782 catalog_manager.cc:4841] TS dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313): tablet df2492a3d98b487f9985576e3fced62b (table test-table [id=c76304518a9c499da9c6e94ac93ebb83]) successfully deleted
Sep 28 21:49:09 dist-test-slave-jrck krb5kdc[4750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:49:09 dist-test-slave-jrck krb5kdc[4750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:09.174933 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:09.168186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.3.0.254:44633)
0928 21:49:09.168299 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:09.168301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:09.168312 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:49:09.168407 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:09.168408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:09.168417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:09.168485 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:09.168994 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.169507 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.170186 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.170367 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.170417 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:09.172508 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:09.172510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:09.172510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:09.172517 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:09.173788 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:09.174077 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:09.174077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:09.174078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:09.174124 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:09.174449 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:09.174450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:09.174451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:09.174575 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:09.174667 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:09.174790 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:49:09.174855 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":67,"threads_started":1}
I20260928 21:49:09.176313 4794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:09.176512 4794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:09.177736 4932 tablet_service.cc:1467] Processing CreateTablet for tablet 224993f467ba4bef8b12fad887508953 (DEFAULT_TABLE table=test-table [id=5cfa2a73bcc44f128337fc90bab4c787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:09.177767 5212 tablet_service.cc:1467] Processing CreateTablet for tablet 224993f467ba4bef8b12fad887508953 (DEFAULT_TABLE table=test-table [id=5cfa2a73bcc44f128337fc90bab4c787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:09.177874 4932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224993f467ba4bef8b12fad887508953. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:09.177887 5212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224993f467ba4bef8b12fad887508953. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:09.178843 5390 tablet_bootstrap.cc:492] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130: Bootstrap starting.
I20260928 21:49:09.178848 5072 tablet_service.cc:1467] Processing CreateTablet for tablet 224993f467ba4bef8b12fad887508953 (DEFAULT_TABLE table=test-table [id=5cfa2a73bcc44f128337fc90bab4c787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:09.178913 5391 tablet_bootstrap.cc:492] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271: Bootstrap starting.
I20260928 21:49:09.178957 5072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224993f467ba4bef8b12fad887508953. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:09.179258 5390 tablet_bootstrap.cc:654] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:09.179258 5391 tablet_bootstrap.cc:654] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:09.179819 5391 tablet_bootstrap.cc:492] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271: No bootstrap required, opened a new log
I20260928 21:49:09.179862 5391 ts_tablet_manager.cc:1397] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:09.179966 5391 raft_consensus.cc:348] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.180023 5391 raft_consensus.cc:374] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:09.180040 5391 raft_consensus.cc:729] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3ee5995204477fbf96843b3dfdc271, State: Initialized, Role: FOLLOWER
I20260928 21:49:09.180079 5391 consensus_queue.cc:260] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.180243 5391 ts_tablet_manager.cc:1428] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:09.180383 5394 tablet_bootstrap.cc:492] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8: Bootstrap starting.
I20260928 21:49:09.180550 5390 tablet_bootstrap.cc:492] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130: No bootstrap required, opened a new log
I20260928 21:49:09.180603 5390 ts_tablet_manager.cc:1397] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:09.180742 5394 tablet_bootstrap.cc:654] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:09.180732 5390 raft_consensus.cc:348] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.180796 5390 raft_consensus.cc:374] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:09.180866 5390 raft_consensus.cc:729] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0afb0ba688418faf1dcabeac181130, State: Initialized, Role: FOLLOWER
I20260928 21:49:09.180941 5390 consensus_queue.cc:260] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.181094 5390 ts_tablet_manager.cc:1428] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:09.181358 5394 tablet_bootstrap.cc:492] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8: No bootstrap required, opened a new log
I20260928 21:49:09.181402 5394 ts_tablet_manager.cc:1397] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:09.181495 5394 raft_consensus.cc:348] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.181548 5394 raft_consensus.cc:374] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:09.181564 5394 raft_consensus.cc:729] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea6f3803fc94348a8d2ca6aaa4350f8, State: Initialized, Role: FOLLOWER
I20260928 21:49:09.181607 5394 consensus_queue.cc:260] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.181696 5394 ts_tablet_manager.cc:1428] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:09.274797 5397 raft_consensus.cc:482] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:09.274904 5397 raft_consensus.cc:504] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.275031 5397 leader_election.cc:288] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723), dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313)
I20260928 21:49:09.275218 4950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224993f467ba4bef8b12fad887508953" candidate_uuid: "dd0afb0ba688418faf1dcabeac181130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3ee5995204477fbf96843b3dfdc271" is_pre_election: true
I20260928 21:49:09.275236 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224993f467ba4bef8b12fad887508953" candidate_uuid: "dd0afb0ba688418faf1dcabeac181130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" is_pre_election: true
I20260928 21:49:09.275321 4950 raft_consensus.cc:2454] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd0afb0ba688418faf1dcabeac181130 in term 0.
I20260928 21:49:09.275317 5092 raft_consensus.cc:2454] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd0afb0ba688418faf1dcabeac181130 in term 0.
I20260928 21:49:09.275496 5164 leader_election.cc:302] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3ee5995204477fbf96843b3dfdc271, dd0afb0ba688418faf1dcabeac181130; no voters:
I20260928 21:49:09.275590 5397 raft_consensus.cc:2781] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:09.275628 5397 raft_consensus.cc:482] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:09.275651 5397 raft_consensus.cc:3037] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:09.276292 5397 raft_consensus.cc:504] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.276407 5397 leader_election.cc:288] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [CANDIDATE]: Term 1 election: Requested vote from peers 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723), dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313)
I20260928 21:49:09.276592 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224993f467ba4bef8b12fad887508953" candidate_uuid: "dd0afb0ba688418faf1dcabeac181130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8"
I20260928 21:49:09.276587 4950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224993f467ba4bef8b12fad887508953" candidate_uuid: "dd0afb0ba688418faf1dcabeac181130" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a3ee5995204477fbf96843b3dfdc271"
I20260928 21:49:09.276660 5092 raft_consensus.cc:3037] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:09.276661 4950 raft_consensus.cc:3037] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:09.277208 5092 raft_consensus.cc:2454] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd0afb0ba688418faf1dcabeac181130 in term 1.
I20260928 21:49:09.277259 4950 raft_consensus.cc:2454] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd0afb0ba688418faf1dcabeac181130 in term 1.
I20260928 21:49:09.277379 5166 leader_election.cc:302] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd0afb0ba688418faf1dcabeac181130, dea6f3803fc94348a8d2ca6aaa4350f8; no voters:
I20260928 21:49:09.277495 5397 raft_consensus.cc:2781] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:09.277639 5397 raft_consensus.cc:686] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 1 LEADER]: Becoming Leader. State: Replica: dd0afb0ba688418faf1dcabeac181130, State: Running, Role: LEADER
I20260928 21:49:09.277686 5397 consensus_queue.cc:237] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.278219 4794 catalog_manager.cc:5495] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd0afb0ba688418faf1dcabeac181130 (127.3.0.195). New cstate: current_term: 1 leader_uuid: "dd0afb0ba688418faf1dcabeac181130" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } health_report { overall_health: HEALTHY } } }
I20260928 21:49:09.297827 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:09.295605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.3.0.195:37875)
0928 21:49:09.295731 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:09.295732 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:09.295739 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:49:09.295759 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:09.295760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:09.295768 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:09.295821 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:09.295951 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.296428 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.297207 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.297417 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.297469 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:09.297599 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:09.297633 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:09.297675 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:09.297728 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 21:49:09.297748 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260928 21:49:09.299665 5092 raft_consensus.cc:1261] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Refusing update from remote peer dd0afb0ba688418faf1dcabeac181130: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:09.299674 4950 raft_consensus.cc:1261] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Refusing update from remote peer dd0afb0ba688418faf1dcabeac181130: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:09.299872 5397 consensus_queue.cc:1035] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:09.299983 5398 consensus_queue.cc:1035] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:09.309329 4792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:09.309418 4792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:09.309947 4792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8: Sending DeleteTablet for 3 replicas of tablet 224993f467ba4bef8b12fad887508953
I20260928 21:49:09.310155 5072 tablet_service.cc:1514] Processing DeleteTablet for tablet 224993f467ba4bef8b12fad887508953 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:09 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:51784
I20260928 21:49:09.310209 4932 tablet_service.cc:1514] Processing DeleteTablet for tablet 224993f467ba4bef8b12fad887508953 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:09 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:35552
I20260928 21:49:09.310243 5380 tablet_replica.cc:330] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8: stopping tablet replica
I20260928 21:49:09.310283 5379 tablet_replica.cc:330] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271: stopping tablet replica
I20260928 21:49:09.310286 5380 raft_consensus.cc:2229] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:09.310319 5379 raft_consensus.cc:2229] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:09.310315 5380 raft_consensus.cc:2258] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:09.310349 5379 raft_consensus.cc:2258] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:09.310353 5212 tablet_service.cc:1514] Processing DeleteTablet for tablet 224993f467ba4bef8b12fad887508953 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:09 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:33990
I20260928 21:49:09.310367 4792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:09.310495 4792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:09.310598 5379 ts_tablet_manager.cc:1905] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:09.310705 5381 tablet_replica.cc:330] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130: stopping tablet replica
I20260928 21:49:09.310771 5381 raft_consensus.cc:2229] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:09.310830 5381 raft_consensus.cc:2258] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:09.311112 5381 ts_tablet_manager.cc:1905] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:09.311625 4932 tablet_service.cc:1467] Processing CreateTablet for tablet 92db1e44a70a435c8034aee1070468f1 (DEFAULT_TABLE table=test-table [id=dff915d1070b46f1ab4c526d37132ef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:09.311633 5212 tablet_service.cc:1467] Processing CreateTablet for tablet 92db1e44a70a435c8034aee1070468f1 (DEFAULT_TABLE table=test-table [id=dff915d1070b46f1ab4c526d37132ef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:09.311724 4932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92db1e44a70a435c8034aee1070468f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:09.311723 5212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92db1e44a70a435c8034aee1070468f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:09.312088 5379 ts_tablet_manager.cc:1918] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:09.312134 5379 log.cc:1196] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/224993f467ba4bef8b12fad887508953
I20260928 21:49:09.312314 5379 ts_tablet_manager.cc:1939] T 224993f467ba4bef8b12fad887508953 P 3a3ee5995204477fbf96843b3dfdc271: Deleting consensus metadata
I20260928 21:49:09.312598 5381 ts_tablet_manager.cc:1918] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:09.312633 5381 log.cc:1196] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/224993f467ba4bef8b12fad887508953
I20260928 21:49:09.312757 5381 ts_tablet_manager.cc:1939] T 224993f467ba4bef8b12fad887508953 P dd0afb0ba688418faf1dcabeac181130: Deleting consensus metadata
I20260928 21:49:09.312903 5391 tablet_bootstrap.cc:492] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271: Bootstrap starting.
I20260928 21:49:09.312938 5390 tablet_bootstrap.cc:492] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130: Bootstrap starting.
I20260928 21:49:09.312952 5380 ts_tablet_manager.cc:1905] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:09.313184 5390 tablet_bootstrap.cc:654] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:09.313211 4782 catalog_manager.cc:4841] TS dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875): tablet 224993f467ba4bef8b12fad887508953 (table test-table [id=5cfa2a73bcc44f128337fc90bab4c787]) successfully deleted
I20260928 21:49:09.313283 5391 tablet_bootstrap.cc:654] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:09.313619 4780 catalog_manager.cc:4841] TS 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723): tablet 224993f467ba4bef8b12fad887508953 (table test-table [id=5cfa2a73bcc44f128337fc90bab4c787]) successfully deleted
I20260928 21:49:09.313634 5072 tablet_service.cc:1467] Processing CreateTablet for tablet 92db1e44a70a435c8034aee1070468f1 (DEFAULT_TABLE table=test-table [id=dff915d1070b46f1ab4c526d37132ef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:09.313730 5072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92db1e44a70a435c8034aee1070468f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:09.313819 5391 tablet_bootstrap.cc:492] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271: No bootstrap required, opened a new log
I20260928 21:49:09.313854 5391 ts_tablet_manager.cc:1397] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:09.313992 5391 raft_consensus.cc:348] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.314044 5391 raft_consensus.cc:374] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:09.314073 5391 raft_consensus.cc:729] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a3ee5995204477fbf96843b3dfdc271, State: Initialized, Role: FOLLOWER
I20260928 21:49:09.314116 5391 consensus_queue.cc:260] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.314191 5391 ts_tablet_manager.cc:1428] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:09.314491 5380 ts_tablet_manager.cc:1918] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:09.314535 5380 log.cc:1196] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/224993f467ba4bef8b12fad887508953
I20260928 21:49:09.314693 5380 ts_tablet_manager.cc:1939] T 224993f467ba4bef8b12fad887508953 P dea6f3803fc94348a8d2ca6aaa4350f8: Deleting consensus metadata
I20260928 21:49:09.314872 5394 tablet_bootstrap.cc:492] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8: Bootstrap starting.
I20260928 21:49:09.315124 4782 catalog_manager.cc:4841] TS dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313): tablet 224993f467ba4bef8b12fad887508953 (table test-table [id=5cfa2a73bcc44f128337fc90bab4c787]) successfully deleted
I20260928 21:49:09.315141 5394 tablet_bootstrap.cc:654] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:09.315335 5390 tablet_bootstrap.cc:492] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130: No bootstrap required, opened a new log
I20260928 21:49:09.315372 5390 ts_tablet_manager.cc:1397] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 21:49:09.315469 5390 raft_consensus.cc:348] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.315526 5390 raft_consensus.cc:374] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:09.315537 5390 raft_consensus.cc:729] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0afb0ba688418faf1dcabeac181130, State: Initialized, Role: FOLLOWER
I20260928 21:49:09.315582 5390 consensus_queue.cc:260] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.315599 5394 tablet_bootstrap.cc:492] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8: No bootstrap required, opened a new log
I20260928 21:49:09.315637 5394 ts_tablet_manager.cc:1397] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:49:09.315662 5390 ts_tablet_manager.cc:1428] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:09.315730 5394 raft_consensus.cc:348] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.315778 5394 raft_consensus.cc:374] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:09.315788 5394 raft_consensus.cc:729] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea6f3803fc94348a8d2ca6aaa4350f8, State: Initialized, Role: FOLLOWER
I20260928 21:49:09.315837 5394 consensus_queue.cc:260] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.315917 5394 ts_tablet_manager.cc:1428] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:09.368471 5395 raft_consensus.cc:482] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:09.368562 5395 raft_consensus.cc:504] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.368686 5395 leader_election.cc:288] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313), dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875)
I20260928 21:49:09.368865 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db1e44a70a435c8034aee1070468f1" candidate_uuid: "3a3ee5995204477fbf96843b3dfdc271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" is_pre_election: true
I20260928 21:49:09.368960 5092 raft_consensus.cc:2454] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3ee5995204477fbf96843b3dfdc271 in term 0.
I20260928 21:49:09.368912 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db1e44a70a435c8034aee1070468f1" candidate_uuid: "3a3ee5995204477fbf96843b3dfdc271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0afb0ba688418faf1dcabeac181130" is_pre_election: true
I20260928 21:49:09.369006 5231 raft_consensus.cc:2454] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a3ee5995204477fbf96843b3dfdc271 in term 0.
I20260928 21:49:09.369127 4886 leader_election.cc:302] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3ee5995204477fbf96843b3dfdc271, dea6f3803fc94348a8d2ca6aaa4350f8; no voters:
I20260928 21:49:09.369223 5395 raft_consensus.cc:2781] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:09.369273 5395 raft_consensus.cc:482] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:09.369293 5395 raft_consensus.cc:3037] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:09.369946 5395 raft_consensus.cc:504] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.370051 5395 leader_election.cc:288] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [CANDIDATE]: Term 1 election: Requested vote from peers dea6f3803fc94348a8d2ca6aaa4350f8 (127.3.0.194:39313), dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875)
I20260928 21:49:09.370188 5092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db1e44a70a435c8034aee1070468f1" candidate_uuid: "3a3ee5995204477fbf96843b3dfdc271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8"
I20260928 21:49:09.370261 5092 raft_consensus.cc:3037] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:09.370216 5231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92db1e44a70a435c8034aee1070468f1" candidate_uuid: "3a3ee5995204477fbf96843b3dfdc271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd0afb0ba688418faf1dcabeac181130"
I20260928 21:49:09.370289 5231 raft_consensus.cc:3037] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:09.370810 5092 raft_consensus.cc:2454] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3ee5995204477fbf96843b3dfdc271 in term 1.
I20260928 21:49:09.370941 5231 raft_consensus.cc:2454] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a3ee5995204477fbf96843b3dfdc271 in term 1.
I20260928 21:49:09.370975 4886 leader_election.cc:302] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a3ee5995204477fbf96843b3dfdc271, dea6f3803fc94348a8d2ca6aaa4350f8; no voters:
I20260928 21:49:09.371049 5395 raft_consensus.cc:2781] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:09.371186 5395 raft_consensus.cc:686] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [term 1 LEADER]: Becoming Leader. State: Replica: 3a3ee5995204477fbf96843b3dfdc271, State: Running, Role: LEADER
I20260928 21:49:09.371253 5395 consensus_queue.cc:237] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } }
I20260928 21:49:09.371950 4792 catalog_manager.cc:5495] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193). New cstate: current_term: 1 leader_uuid: "3a3ee5995204477fbf96843b3dfdc271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a3ee5995204477fbf96843b3dfdc271" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 } health_report { overall_health: UNKNOWN } } }
I20260928 21:49:09.392908 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:09.390359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.3.0.193:38723)
0928 21:49:09.390519 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:09.390521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:09.390526 (+ 5us) server_negotiation.cc:407] Connection header received
0928 21:49:09.390559 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:09.390559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:09.390577 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:09.390647 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:09.390758 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.391243 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.391953 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.392198 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.392259 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:09.392434 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:09.392606 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:09.392718 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:09.392773 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 21:49:09.392797 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260928 21:49:09.397279 5415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:09.395129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:56677 (local address 127.3.0.194:39313)
0928 21:49:09.395219 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:09.395221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:09.395228 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:49:09.395249 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:09.395250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:09.395259 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:09.395317 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:09.395490 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.395939 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.396744 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.396933 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.396980 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:09.397110 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:09.397134 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:09.397164 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:09.397205 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 21:49:09.397220 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":49,"threads_started":1}
I20260928 21:49:09.397375 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:09.395006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39313 (local address 127.3.0.193:56677)
0928 21:49:09.395145 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:09.395155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:09.395194 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:09.395373 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:09.395374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:09.395376 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:09.395433 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:09.395434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.396009 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.396009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:09.396687 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:09.396690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.396994 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.396995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:09.397042 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:09.397055 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:09.397174 (+ 119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:09.397175 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:49:09.397234 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:49:09.397267 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
I20260928 21:49:09.398996 5231 raft_consensus.cc:1261] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Refusing update from remote peer 3a3ee5995204477fbf96843b3dfdc271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:09.399025 5092 raft_consensus.cc:1261] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Refusing update from remote peer 3a3ee5995204477fbf96843b3dfdc271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:09.399238 5409 consensus_queue.cc:1035] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd0afb0ba688418faf1dcabeac181130" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:09.399366 5395 consensus_queue.cc:1035] T 92db1e44a70a435c8034aee1070468f1 P 3a3ee5995204477fbf96843b3dfdc271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea6f3803fc94348a8d2ca6aaa4350f8" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:09.405676 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:09.403549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:42209 (local address 127.3.0.193:38723)
0928 21:49:09.403585 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:09.403586 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:09.403631 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:49:09.403696 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:09.403697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:09.403706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:09.403758 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:09.403858 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.404292 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.405000 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.405256 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.405316 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:09.405473 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:09.405503 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:09.405546 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:09.405613 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 21:49:09.405632 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:49:09.405738 5419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:09.403442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:38723 (local address 127.3.0.194:42209)
0928 21:49:09.403582 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:09.403592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:09.403645 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:09.403761 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:09.403762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:09.403763 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:09.403808 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:09.403809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.404339 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.404340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:09.404932 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:09.404934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:09.405333 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:09.405334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:09.405405 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:09.405411 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:09.405569 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:09.405571 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:49:09.405606 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 21:49:09.405652 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
W20260928 21:49:09.406018 5364 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:49:10.923314 4793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:10.923415 4793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:10.924198 4793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc72d04ef78041fab55cab014157b2f8: Sending DeleteTablet for 3 replicas of tablet 92db1e44a70a435c8034aee1070468f1
I20260928 21:49:10.924412 4932 tablet_service.cc:1514] Processing DeleteTablet for tablet 92db1e44a70a435c8034aee1070468f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:10 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:35552
I20260928 21:49:10.924508 5072 tablet_service.cc:1514] Processing DeleteTablet for tablet 92db1e44a70a435c8034aee1070468f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:10 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:51784
I20260928 21:49:10.924571 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 4867
I20260928 21:49:10.924722 5212 tablet_service.cc:1514] Processing DeleteTablet for tablet 92db1e44a70a435c8034aee1070468f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:10 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:33990
I20260928 21:49:10.925280 5445 tablet_replica.cc:330] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130: stopping tablet replica
I20260928 21:49:10.925377 5445 raft_consensus.cc:2229] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:10.925428 5445 raft_consensus.cc:2258] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:10.925674 5445 ts_tablet_manager.cc:1905] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:10.927235 5445 ts_tablet_manager.cc:1918] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:49:10.927296 5445 log.cc:1196] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/92db1e44a70a435c8034aee1070468f1
I20260928 21:49:10.927485 5445 ts_tablet_manager.cc:1939] T 92db1e44a70a435c8034aee1070468f1 P dd0afb0ba688418faf1dcabeac181130: Deleting consensus metadata
I20260928 21:49:10.927856 4782 catalog_manager.cc:4841] TS dd0afb0ba688418faf1dcabeac181130 (127.3.0.195:37875): tablet 92db1e44a70a435c8034aee1070468f1 (table test-table [id=dff915d1070b46f1ab4c526d37132ef2]) successfully deleted
I20260928 21:49:10.930142 5444 tablet_replica.cc:330] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8: stopping tablet replica
I20260928 21:49:10.930233 5444 raft_consensus.cc:2229] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:10.930284 5444 raft_consensus.cc:2258] T 92db1e44a70a435c8034aee1070468f1 P dea6f3803fc94348a8d2ca6aaa4350f8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 21:49:10.930297 4780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:38723 (error 108)
I20260928 21:49:10.930464 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 5007
I20260928 21:49:10.935377 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:10.930656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:38723 (local address 127.0.0.1:35564)
0928 21:49:10.935277 (+ 4621us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:10.935332 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:38723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4593,"thread_start_us":57,"threads_started":1}
I20260928 21:49:10.935604 5447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:10.935515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39313 (local address 127.0.0.1:51824)
0928 21:49:10.935565 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:10.935587 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.194:39313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 21:49:10.935626 4780 catalog_manager.cc:4584] TS 3a3ee5995204477fbf96843b3dfdc271 (127.3.0.193:38723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92db1e44a70a435c8034aee1070468f1: Network error: Client connection negotiation failed: client connection to 127.3.0.193:38723: connect: Connection refused (error 111)
I20260928 21:49:10.935719 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 5147
I20260928 21:49:10.940191 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 4763
2026-09-28T21:49:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5950 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:10 dist-test-slave-jrck krb5kdc[5450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:10 dist-test-slave-jrck krb5kdc[5450](info): set up 2 sockets
Sep 28 21:49:10 dist-test-slave-jrck krb5kdc[5450](info): commencing operation
krb5kdc: starting...
W20260928 21:49:11.895932 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:11 dist-test-slave-jrck krb5kdc[5450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:12.001073 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:46467
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:12.081537 5463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:12.081732 5463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:12.081761 5463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:12.082885 5463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:12.082926 5463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:12.082944 5463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:12.082966 5463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:12.082990 5463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:12.084430 5463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:46467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:12.084694 5463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:12.084889 5463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:12.089452 5468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:12.089516 5471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:12.089612 5463 server_base.cc:1034] running on GCE node
W20260928 21:49:12.089452 5469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:12.089919 5463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:12.090210 5463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:12.091354 5463 hybrid_clock.cc:648] HybridClock initialized: now 1790632152091334 us; error 44 us; skew 500 ppm
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:12.093655 5463 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:49:12.094101 5463 webserver.cc:460] Webserver started at http://127.3.0.254:45663/ using document root <none> and password file <none>
I20260928 21:49:12.094316 5463 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:12.094362 5463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:12.094475 5463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:12.095450 5463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "66ef7aea71294c379ad1df7102c22b40"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "e947037a1c77ecff27482859f5fa9342"
server_key_iv: "fd198537f46528a4b8415b45672dd6d5"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.095758 5463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "66ef7aea71294c379ad1df7102c22b40"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "e947037a1c77ecff27482859f5fa9342"
server_key_iv: "fd198537f46528a4b8415b45672dd6d5"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.097004 5463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:12.097640 5478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:12.097795 5463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:12.097885 5463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "66ef7aea71294c379ad1df7102c22b40"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "e947037a1c77ecff27482859f5fa9342"
server_key_iv: "fd198537f46528a4b8415b45672dd6d5"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.097941 5463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:12.121249 5463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:12.122983 5463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:12.123119 5463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:12.126848 5463 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:46467
I20260928 21:49:12.126888 5530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:46467 every 8 connection(s)
I20260928 21:49:12.127205 5463 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:12.128022 5531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:12.130219 5531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40: Bootstrap starting.
I20260928 21:49:12.130757 5531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:12.131013 5531 log.cc:824] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:12.131618 5531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40: No bootstrap required, opened a new log
I20260928 21:49:12.133179 5531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ef7aea71294c379ad1df7102c22b40" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46467 } }
I20260928 21:49:12.133316 5531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:12.133338 5531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ef7aea71294c379ad1df7102c22b40, State: Initialized, Role: FOLLOWER
I20260928 21:49:12.133487 5531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ef7aea71294c379ad1df7102c22b40" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46467 } }
I20260928 21:49:12.133548 5531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:12.133569 5531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:12.133613 5531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:12.134087 5531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ef7aea71294c379ad1df7102c22b40" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46467 } }
I20260928 21:49:12.134171 5531 leader_election.cc:302] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66ef7aea71294c379ad1df7102c22b40; no voters:
I20260928 21:49:12.134317 5531 leader_election.cc:288] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:12.134377 5534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:12.134564 5531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:12.134673 5534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [term 1 LEADER]: Becoming Leader. State: Replica: 66ef7aea71294c379ad1df7102c22b40, State: Running, Role: LEADER
I20260928 21:49:12.134748 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 5463
I20260928 21:49:12.134845 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:12.134815 5534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ef7aea71294c379ad1df7102c22b40" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46467 } }
I20260928 21:49:12.135017 3075 external_mini_cluster.cc:1428] Setting key c36d2950365dc6d50d620273dfd0b968
I20260928 21:49:12.135138 5535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66ef7aea71294c379ad1df7102c22b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ef7aea71294c379ad1df7102c22b40" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46467 } } }
I20260928 21:49:12.135143 5536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ef7aea71294c379ad1df7102c22b40. Latest consensus state: current_term: 1 leader_uuid: "66ef7aea71294c379ad1df7102c22b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ef7aea71294c379ad1df7102c22b40" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46467 } } }
I20260928 21:49:12.135219 5535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:12.135223 5536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:12.136262 5543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:12.136777 5543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:12.138231 5543 catalog_manager.cc:1347] Generated new cluster ID: 2f3629e141794a7aa5561feb89b6cf95
I20260928 21:49:12.138281 5543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:12.146407 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:12.135597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.3.0.254:46467)
0928 21:49:12.136316 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:12.136326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:12.136365 (+ 39us) server_negotiation.cc:407] Connection header received
0928 21:49:12.136500 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:12.136508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:12.136569 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:12.136707 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:12.137435 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.138573 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.139608 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.140216 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.140316 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:12.142940 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:12.142946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:12.142946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:12.142960 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:12.144831 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:12.145176 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:12.145179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:12.145180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:12.145310 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:12.145521 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:12.145522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:12.145522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:12.145837 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:12.145996 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:12.146060 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 21:49:12.146192 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":67,"threads_started":1}
I20260928 21:49:12.148816 5543 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:12.149263 5543 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:12.156716 5543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66ef7aea71294c379ad1df7102c22b40: Generated new TSK 0
I20260928 21:49:12.156864 5543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:49:12.184667 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46467
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:12.264233 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:12.264446 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:12.264499 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:12.265690 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:12.265733 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:12.265761 5557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:12.267158 5557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:12.267373 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:12.267570 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:12.272171 5565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:12.272200 5563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:12.272171 5562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:12.272198 5557 server_base.cc:1034] running on GCE node
I20260928 21:49:12.272583 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:12.272823 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:12.273960 5557 hybrid_clock.cc:648] HybridClock initialized: now 1790632152273953 us; error 25 us; skew 500 ppm
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:12.276216 5557 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:49:12.276643 5557 webserver.cc:460] Webserver started at http://127.3.0.193:42589/ using document root <none> and password file <none>
I20260928 21:49:12.276835 5557 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:12.276882 5557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:12.277014 5557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:12.278005 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "88ea90d5c9374a4bb1bf1a0e188e343e"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "d34d5b098a6606dd10f78bfe8006ada0"
server_key_iv: "3bbfc164f7043bbba90d11d3ab7d1ba3"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.278333 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "88ea90d5c9374a4bb1bf1a0e188e343e"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "d34d5b098a6606dd10f78bfe8006ada0"
server_key_iv: "3bbfc164f7043bbba90d11d3ab7d1ba3"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.279630 5557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:12.280277 5572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:12.280421 5557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:12.280517 5557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "88ea90d5c9374a4bb1bf1a0e188e343e"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "d34d5b098a6606dd10f78bfe8006ada0"
server_key_iv: "3bbfc164f7043bbba90d11d3ab7d1ba3"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.280582 5557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:12.297139 5557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:12.298874 5557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:12.299002 5557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:12.299197 5557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:12.299472 5557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:12.299504 5557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:12.299530 5557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:12.299552 5557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:12.304893 5557 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:42449
I20260928 21:49:12.304957 5685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:42449 every 8 connection(s)
I20260928 21:49:12.305217 5557 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:49:12.309092 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 5557
I20260928 21:49:12.309183 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:12.309350 3075 external_mini_cluster.cc:1428] Setting key f9677123a04c2cf73adda1d4aa2c878a
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:12.313978 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:12.305836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:44333 (local address 127.3.0.254:46467)
0928 21:49:12.305885 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:12.305887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:12.306077 (+ 190us) server_negotiation.cc:407] Connection header received
0928 21:49:12.306606 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:12.306608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:12.306624 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:12.306669 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:12.307584 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.308077 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.309148 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.309439 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.309497 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:12.311480 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:12.311482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:12.311482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:12.311489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:12.312785 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:12.313136 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:12.313138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:12.313138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:12.313193 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:12.313397 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:12.313398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:12.313399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:12.313507 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:12.313562 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:12.313807 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 21:49:12.313892 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:49:12.314137 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:12.305801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46467 (local address 127.3.0.193:44333)
0928 21:49:12.305967 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:12.306004 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:12.306487 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:12.306752 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:12.306756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:12.307304 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:12.307488 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:12.307493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.308159 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.308160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:12.309069 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:12.309072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.309520 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.309521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:12.309639 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:12.310118 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:12.310126 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:12.311398 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:12.312862 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:12.312869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:12.312871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:12.313072 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:12.313244 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:12.313245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:12.313245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:12.313342 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:12.313559 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:12.313564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:12.313694 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 21:49:12.313778 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:49:12.313935 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260928 21:49:12.314536 5686 heartbeater.cc:347] Connected to a master server at 127.3.0.254:46467
I20260928 21:49:12.314625 5686 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:12.314783 5686 heartbeater.cc:511] Master 127.3.0.254:46467 requested a full tablet report, sending...
I20260928 21:49:12.315248 5495 ts_manager.cc:194] Registered new tserver with Master: 88ea90d5c9374a4bb1bf1a0e188e343e (127.3.0.193:42449)
I20260928 21:49:12.315824 5495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:44333
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:49:12.330394 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46467
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:12.408267 5691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:12.408494 5691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:12.408517 5691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:12.409638 5691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:12.409672 5691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:12.409698 5691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:12.411089 5691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:12.411273 5691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:12.411448 5691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:12.416075 5697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:12.416081 5696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:12.416093 5699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:12.416174 5691 server_base.cc:1034] running on GCE node
I20260928 21:49:12.416446 5691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:12.416721 5691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:12.417860 5691 hybrid_clock.cc:648] HybridClock initialized: now 1790632152417851 us; error 26 us; skew 500 ppm
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:12.420164 5691 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:49:12.420601 5691 webserver.cc:460] Webserver started at http://127.3.0.194:39799/ using document root <none> and password file <none>
I20260928 21:49:12.420778 5691 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:12.420815 5691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:12.420907 5691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:12.421891 5691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "d2f5b66b9bec43cb937575064891755a"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "15ec877ea1041caa3837dbf7444e8135"
server_key_iv: "bd694f4ac216f784a13a32e295ad577a"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.422173 5691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "d2f5b66b9bec43cb937575064891755a"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "15ec877ea1041caa3837dbf7444e8135"
server_key_iv: "bd694f4ac216f784a13a32e295ad577a"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.423327 5691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:49:12.423954 5706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:12.424084 5691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:12.424163 5691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "d2f5b66b9bec43cb937575064891755a"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "15ec877ea1041caa3837dbf7444e8135"
server_key_iv: "bd694f4ac216f784a13a32e295ad577a"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.424220 5691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:12.435256 5691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:12.437234 5691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:12.437361 5691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:12.437556 5691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:12.437834 5691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:12.437866 5691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:12.437887 5691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:12.437911 5691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:12.443096 5691 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:35647
I20260928 21:49:12.443135 5819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:35647 every 8 connection(s)
I20260928 21:49:12.443425 5691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:49:12.444584 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 5691
I20260928 21:49:12.444661 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:12.444828 3075 external_mini_cluster.cc:1428] Setting key 3fc6ad548b2e3680121df1dd6e64ab1f
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
I20260928 21:49:12.453001 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:12.444064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:36051 (local address 127.3.0.254:46467)
0928 21:49:12.444129 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:12.444131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:12.444320 (+ 189us) server_negotiation.cc:407] Connection header received
0928 21:49:12.444848 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:12.444850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:12.444862 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:12.444948 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:12.446166 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.446690 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.447689 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.447971 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.448033 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:12.450257 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:12.450259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:12.450259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:12.450268 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:12.451666 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:12.452038 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:12.452039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:12.452040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:12.452101 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:12.452308 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:12.452309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:12.452309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:12.452507 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:12.452583 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:12.452814 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 21:49:12.452903 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:49:12.453128 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:12.444032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46467 (local address 127.3.0.194:36051)
0928 21:49:12.444199 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:12.444238 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:12.444757 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:12.445074 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:12.445077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:12.445781 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:12.446067 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:12.446074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.446783 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.446785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:12.447613 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:12.447615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.448044 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.448045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:12.448140 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:12.448664 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:12.448675 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:12.450164 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:12.451739 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:12.451745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:12.451747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:12.451970 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:12.452163 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:12.452164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:12.452164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:12.452252 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:12.452572 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:12.452575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:12.452692 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 21:49:12.452774 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:49:12.452938 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260928 21:49:12.453594 5820 heartbeater.cc:347] Connected to a master server at 127.3.0.254:46467
I20260928 21:49:12.453706 5820 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:12.453886 5820 heartbeater.cc:511] Master 127.3.0.254:46467 requested a full tablet report, sending...
I20260928 21:49:12.454255 5495 ts_manager.cc:194] Registered new tserver with Master: d2f5b66b9bec43cb937575064891755a (127.3.0.194:35647)
I20260928 21:49:12.454629 5495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:36051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:49:12.467236 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46467
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:37849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:12.545737 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:12.545939 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:12.545960 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:12.547072 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:12.547111 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:12.547135 5825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:12.548555 5825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:37849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:12.548741 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:12.548918 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:49:12.553718 5825 server_base.cc:1034] running on GCE node
W20260928 21:49:12.553818 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:12.553861 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:12.553872 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:12.554204 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:12.554469 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:12.555611 5825 hybrid_clock.cc:648] HybridClock initialized: now 1790632152555599 us; error 29 us; skew 500 ppm
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:12.557916 5825 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:49:12.558333 5825 webserver.cc:460] Webserver started at http://127.3.0.195:39223/ using document root <none> and password file <none>
I20260928 21:49:12.558557 5825 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:12.558604 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:12.558722 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:12.559724 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "165926f4fc204c02a77700f8a1d2ce9a"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "fbe5a612232822d811094d911fb212e4"
server_key_iv: "614b01c48a4682fa0b0d9359a97ef705"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.560045 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "165926f4fc204c02a77700f8a1d2ce9a"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "fbe5a612232822d811094d911fb212e4"
server_key_iv: "614b01c48a4682fa0b0d9359a97ef705"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.561302 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:49:12.561942 5840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:12.562081 5825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:12.562161 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "165926f4fc204c02a77700f8a1d2ce9a"
format_stamp: "Formatted at 2026-09-28 21:49:12 on dist-test-slave-jrck"
server_key: "fbe5a612232822d811094d911fb212e4"
server_key_iv: "614b01c48a4682fa0b0d9359a97ef705"
server_key_version: "encryptionkey@0"
I20260928 21:49:12.562219 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:12.588420 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:12.590138 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:12.590273 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:12.590495 5825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:12.590775 5825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:12.590808 5825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:12.590837 5825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:12.590858 5825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:12.596230 5825 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:33195
I20260928 21:49:12.596283 5953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:33195 every 8 connection(s)
I20260928 21:49:12.596613 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:49:12.601516 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 5825
I20260928 21:49:12.601588 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:12.601748 3075 external_mini_cluster.cc:1428] Setting key d1cf8c38090208f23b2367bb359838ce
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632152, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:12.605104 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:12.597282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:33339 (local address 127.3.0.254:46467)
0928 21:49:12.597367 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:12.597369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:12.597500 (+ 131us) server_negotiation.cc:407] Connection header received
0928 21:49:12.597983 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:12.597984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:12.597993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:12.598038 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:12.598893 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.599371 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.600385 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.600575 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.600617 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:12.602840 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:12.602842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:12.602842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:12.602849 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:12.603890 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:12.604247 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:12.604248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:12.604248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:12.604302 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:12.604530 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:12.604531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:12.604532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:12.604636 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:12.604690 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:12.604944 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 21:49:12.605018 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:49:12.605264 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:12.597258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46467 (local address 127.3.0.195:33339)
0928 21:49:12.597392 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:12.597428 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:12.597893 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:12.598100 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:12.598104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:12.598639 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:12.598828 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:12.598833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.599434 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.599435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:12.600312 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:12.600315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:12.600634 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:12.600634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:12.600734 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:12.601212 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:12.601220 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:12.602763 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:12.603962 (+ 1199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:12.603968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:12.603970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:12.604167 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:12.604353 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:12.604354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:12.604354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:12.604481 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:12.604689 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:12.604694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:12.604825 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 21:49:12.604916 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:49:12.605062 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20260928 21:49:12.605649 5954 heartbeater.cc:347] Connected to a master server at 127.3.0.254:46467
I20260928 21:49:12.605746 5954 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:12.605907 5954 heartbeater.cc:511] Master 127.3.0.254:46467 requested a full tablet report, sending...
I20260928 21:49:12.606241 5495 ts_manager.cc:194] Registered new tserver with Master: 165926f4fc204c02a77700f8a1d2ce9a (127.3.0.195:33195)
I20260928 21:49:12.606600 5495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:33339
I20260928 21:49:12.613304 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:49:12.623553 5547 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:49:12.622683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.3.0.254:46467)
0928 21:49:12.622743 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:12.622745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:12.622833 (+ 88us) server_negotiation.cc:407] Connection header received
0928 21:49:12.622903 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:12.622903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:12.622914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:12.622961 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:12.623482 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50464: BlockingRecv error: recv got EOF from 127.0.0.1:50464 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:49:12.624389 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 5557
I20260928 21:49:12.628348 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 5691
I20260928 21:49:12.631898 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 5825
I20260928 21:49:12.635730 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 5463
2026-09-28T21:49:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1694 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5966](info): set up 2 sockets
Sep 28 21:49:12 dist-test-slave-jrck krb5kdc[5966](info): commencing operation
krb5kdc: starting...
W20260928 21:49:13.580037 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:13 dist-test-slave-jrck krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:13.688316 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:45117
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:42257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:45117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:13.766155 5979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:13.766383 5979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:13.766417 5979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:13.767578 5979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:13.767627 5979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:13.767645 5979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:13.767660 5979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:13.767676 5979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:13.769243 5979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:45117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:45117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:13.769486 5979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:13.769690 5979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:13.774192 5985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:13.774228 5987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:13.774277 5979 server_base.cc:1034] running on GCE node
W20260928 21:49:13.774183 5984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:13.774657 5979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:13.774991 5979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:13.776134 5979 hybrid_clock.cc:648] HybridClock initialized: now 1790632153776118 us; error 28 us; skew 500 ppm
Sep 28 21:49:13 dist-test-slave-jrck krb5kdc[5966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:13.778457 5979 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:49:13.778858 5979 webserver.cc:460] Webserver started at http://127.3.0.254:36347/ using document root <none> and password file <none>
I20260928 21:49:13.779055 5979 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:13.779102 5979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:13.779225 5979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:13.780284 5979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "249d223814014294b7b07dbb1a94281c"
format_stamp: "Formatted at 2026-09-28 21:49:13 on dist-test-slave-jrck"
server_key: "12553ef3326b6e0aeadce3611eab6d1f"
server_key_iv: "d1fea18384297ed2ebbc872b2bf1edfb"
server_key_version: "encryptionkey@0"
I20260928 21:49:13.780635 5979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "249d223814014294b7b07dbb1a94281c"
format_stamp: "Formatted at 2026-09-28 21:49:13 on dist-test-slave-jrck"
server_key: "12553ef3326b6e0aeadce3611eab6d1f"
server_key_iv: "d1fea18384297ed2ebbc872b2bf1edfb"
server_key_version: "encryptionkey@0"
I20260928 21:49:13.781937 5979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:13.782629 5994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:13.782783 5979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:13.782866 5979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "249d223814014294b7b07dbb1a94281c"
format_stamp: "Formatted at 2026-09-28 21:49:13 on dist-test-slave-jrck"
server_key: "12553ef3326b6e0aeadce3611eab6d1f"
server_key_iv: "d1fea18384297ed2ebbc872b2bf1edfb"
server_key_version: "encryptionkey@0"
I20260928 21:49:13.782934 5979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:13.793221 5979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:13.795002 5979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:13.795137 5979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:13.798895 5979 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:45117
I20260928 21:49:13.798928 6046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:45117 every 8 connection(s)
I20260928 21:49:13.799263 5979 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:13.800098 6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:13.802338 6047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c: Bootstrap starting.
I20260928 21:49:13.802423 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 5979
I20260928 21:49:13.802501 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:13.802651 3075 external_mini_cluster.cc:1428] Setting key 387f14d918414420c0f6c94b34814735
I20260928 21:49:13.802944 6047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:13.803267 6047 log.cc:824] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:13.803895 6047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c: No bootstrap required, opened a new log
I20260928 21:49:13.805987 6047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249d223814014294b7b07dbb1a94281c" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45117 } }
I20260928 21:49:13.806157 6047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:13.806188 6047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249d223814014294b7b07dbb1a94281c, State: Initialized, Role: FOLLOWER
I20260928 21:49:13.806311 6047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249d223814014294b7b07dbb1a94281c" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45117 } }
I20260928 21:49:13.806377 6047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:13.806411 6047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:13.806454 6047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:13.806988 6047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249d223814014294b7b07dbb1a94281c" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45117 } }
I20260928 21:49:13.807096 6047 leader_election.cc:302] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 249d223814014294b7b07dbb1a94281c; no voters:
I20260928 21:49:13.807257 6047 leader_election.cc:288] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:13.807303 6052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:13.807484 6052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [term 1 LEADER]: Becoming Leader. State: Replica: 249d223814014294b7b07dbb1a94281c, State: Running, Role: LEADER
I20260928 21:49:13.807504 6047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:13.807590 6052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249d223814014294b7b07dbb1a94281c" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45117 } }
I20260928 21:49:13.807979 6052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 249d223814014294b7b07dbb1a94281c. Latest consensus state: current_term: 1 leader_uuid: "249d223814014294b7b07dbb1a94281c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249d223814014294b7b07dbb1a94281c" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45117 } } }
I20260928 21:49:13.808204 6052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:13.808142 6053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "249d223814014294b7b07dbb1a94281c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "249d223814014294b7b07dbb1a94281c" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45117 } } }
I20260928 21:49:13.808640 6053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:13.808789 6061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 21:49:13 dist-test-slave-jrck krb5kdc[5966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:13.809386 6061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:13.811691 6061 catalog_manager.cc:1347] Generated new cluster ID: 4578b26216ed47e4b802f5f269d96fe8
I20260928 21:49:13.811743 6061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:49:13.812984 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:13.803099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.3.0.254:45117)
0928 21:49:13.803277 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:13.803286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:13.803313 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:49:13.803727 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:13.803734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:13.803783 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:13.803900 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:13.804644 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:13.805514 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:13.806533 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:13.806949 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:13.807030 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:13.809721 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:13.809724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:13.809725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:13.809735 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:13.811563 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:13.811915 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:13.811932 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:13.811933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:13.812019 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:13.812195 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:13.812196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:13.812196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:13.812420 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:13.812612 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:13.812682 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 21:49:13.812808 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260928 21:49:13.813900 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:49:13.824606 6061 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:13.825295 6061 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:13.826278 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:13.822952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.3.0.254:45117)
0928 21:49:13.823004 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:13.823007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:13.823122 (+ 115us) server_negotiation.cc:407] Connection header received
0928 21:49:13.823182 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:13.823183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:13.823193 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:13.823239 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:13.823605 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:13.824089 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:13.824971 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:13.825181 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:13.825237 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:13.825853 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:13.825855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:13.825855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:13.826027 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:13.826042 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:13.826048 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:13.826093 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:13.826164 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 21:49:13.826204 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:49:13.837478 6061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 249d223814014294b7b07dbb1a94281c: Generated new TSK 0
I20260928 21:49:13.837692 6061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:49:13.840072 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 5979
2026-09-28T21:49:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1202 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:13 dist-test-slave-jrck krb5kdc[6080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:13 dist-test-slave-jrck krb5kdc[6080](info): set up 2 sockets
Sep 28 21:49:13 dist-test-slave-jrck krb5kdc[6080](info): commencing operation
krb5kdc: starting...
W20260928 21:49:14.787950 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:14 dist-test-slave-jrck krb5kdc[6080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:14.890563 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33651
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:33651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:14.967200 6093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:14.967411 6093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:14.967432 6093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:14.968545 6093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:14.968580 6093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:14.968592 6093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:14.968603 6093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:14.968614 6093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:14.970067 6093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:33651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:14.970259 6093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:14.970430 6093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:14.974812 6098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:14.974855 6101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:14.974813 6099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:14.975042 6093 server_base.cc:1034] running on GCE node
I20260928 21:49:14.975275 6093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:14.975543 6093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:14.976689 6093 hybrid_clock.cc:648] HybridClock initialized: now 1790632154976673 us; error 28 us; skew 500 ppm
Sep 28 21:49:14 dist-test-slave-jrck krb5kdc[6080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632154, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:14.978969 6093 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:49:14.979419 6093 webserver.cc:460] Webserver started at http://127.3.0.254:38791/ using document root <none> and password file <none>
I20260928 21:49:14.979620 6093 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:14.979667 6093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:14.979794 6093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:14.980890 6093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "33bca5ff83604cbdb07615692316b77a"
format_stamp: "Formatted at 2026-09-28 21:49:14 on dist-test-slave-jrck"
server_key: "05b52e681aa3b977b0c0f647d9e07065"
server_key_iv: "421aa6f3bc5cebc420f45e12247f9ad1"
server_key_version: "encryptionkey@0"
I20260928 21:49:14.981207 6093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "33bca5ff83604cbdb07615692316b77a"
format_stamp: "Formatted at 2026-09-28 21:49:14 on dist-test-slave-jrck"
server_key: "05b52e681aa3b977b0c0f647d9e07065"
server_key_iv: "421aa6f3bc5cebc420f45e12247f9ad1"
server_key_version: "encryptionkey@0"
I20260928 21:49:14.982381 6093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:14.983019 6108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:14.983160 6093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:14.983239 6093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "33bca5ff83604cbdb07615692316b77a"
format_stamp: "Formatted at 2026-09-28 21:49:14 on dist-test-slave-jrck"
server_key: "05b52e681aa3b977b0c0f647d9e07065"
server_key_iv: "421aa6f3bc5cebc420f45e12247f9ad1"
server_key_version: "encryptionkey@0"
I20260928 21:49:14.983296 6093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:15.000070 6093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:15.001868 6093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:15.002004 6093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:15.005784 6093 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:33651
I20260928 21:49:15.005820 6160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:33651 every 8 connection(s)
I20260928 21:49:15.006132 6093 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:15.007021 6161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:15.009277 6161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a: Bootstrap starting.
I20260928 21:49:15.009819 6161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:15.010071 6161 log.cc:824] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:15.010706 6161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a: No bootstrap required, opened a new log
I20260928 21:49:15.012238 6161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bca5ff83604cbdb07615692316b77a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33651 } }
I20260928 21:49:15.012372 6161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:15.012393 6161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33bca5ff83604cbdb07615692316b77a, State: Initialized, Role: FOLLOWER
I20260928 21:49:15.012540 6161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bca5ff83604cbdb07615692316b77a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33651 } }
I20260928 21:49:15.012614 6161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:15.012637 6161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:15.012683 6161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:15.013433 6161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bca5ff83604cbdb07615692316b77a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33651 } }
I20260928 21:49:15.013521 6161 leader_election.cc:302] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33bca5ff83604cbdb07615692316b77a; no voters:
I20260928 21:49:15.013667 6161 leader_election.cc:288] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:15.013741 6164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:15.013898 6161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:15.013916 6164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [term 1 LEADER]: Becoming Leader. State: Replica: 33bca5ff83604cbdb07615692316b77a, State: Running, Role: LEADER
I20260928 21:49:15.013998 6164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bca5ff83604cbdb07615692316b77a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33651 } }
I20260928 21:49:15.014371 6166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33bca5ff83604cbdb07615692316b77a. Latest consensus state: current_term: 1 leader_uuid: "33bca5ff83604cbdb07615692316b77a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bca5ff83604cbdb07615692316b77a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33651 } } }
I20260928 21:49:15.014461 6166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:15.014370 6165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33bca5ff83604cbdb07615692316b77a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33bca5ff83604cbdb07615692316b77a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33651 } } }
I20260928 21:49:15.014737 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 6093
I20260928 21:49:15.014828 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:15.014696 6165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:15.015007 3075 external_mini_cluster.cc:1428] Setting key 2f9f04423089935d9aeadc6df3ca5a4f
W20260928 21:49:15.015636 6180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:49:15.015689 6180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:49:15.015729 6173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:15.015875 6173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:15.017292 6173 catalog_manager.cc:1347] Generated new cluster ID: 198a85255e474f968a26d1c9e93587f7
I20260928 21:49:15.017340 6173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:15.025913 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:15.015510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.3.0.254:33651)
0928 21:49:15.015734 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:15.015743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:15.015774 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:49:15.015884 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:15.015890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:15.015932 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:15.016047 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:15.016713 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.018084 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.019108 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.019740 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.019855 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:15.022507 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:15.022511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:15.022511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:15.022522 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:15.024354 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:15.024767 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:15.024770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:15.024770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:15.024868 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:15.025049 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:15.025050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:15.025050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:15.025323 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:15.025459 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:15.025566 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:49:15.025692 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260928 21:49:15.030742 6173 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:15.031278 6173 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:15.033160 6173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33bca5ff83604cbdb07615692316b77a: Generated new TSK 0
I20260928 21:49:15.033317 6173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:49:15.056162 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33651
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:15.133773 6187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:15.133972 6187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:15.133994 6187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:15.135142 6187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:15.135183 6187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:15.135208 6187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:15.136636 6187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:15.136849 6187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:15.137049 6187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:15.141834 6193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:15.141830 6192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:15.141897 6195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:15.142046 6187 server_base.cc:1034] running on GCE node
I20260928 21:49:15.142282 6187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:15.142506 6187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:15.143646 6187 hybrid_clock.cc:648] HybridClock initialized: now 1790632155143626 us; error 32 us; skew 500 ppm
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632155, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:15.145941 6187 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:49:15.146330 6187 webserver.cc:460] Webserver started at http://127.3.0.193:35969/ using document root <none> and password file <none>
I20260928 21:49:15.146519 6187 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:15.146565 6187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:15.146685 6187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:15.147689 6187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "4fa81a9cf2a346e48c46de6aadc360a3"
format_stamp: "Formatted at 2026-09-28 21:49:15 on dist-test-slave-jrck"
server_key: "a729ff4e9b8ac4e39fa1cb64ce427117"
server_key_iv: "1c7ebc53ac1db38d92b6447c0bbcebf7"
server_key_version: "encryptionkey@0"
I20260928 21:49:15.148016 6187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "4fa81a9cf2a346e48c46de6aadc360a3"
format_stamp: "Formatted at 2026-09-28 21:49:15 on dist-test-slave-jrck"
server_key: "a729ff4e9b8ac4e39fa1cb64ce427117"
server_key_iv: "1c7ebc53ac1db38d92b6447c0bbcebf7"
server_key_version: "encryptionkey@0"
I20260928 21:49:15.149250 6187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:49:15.149904 6202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:15.150091 6187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:15.150183 6187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "4fa81a9cf2a346e48c46de6aadc360a3"
format_stamp: "Formatted at 2026-09-28 21:49:15 on dist-test-slave-jrck"
server_key: "a729ff4e9b8ac4e39fa1cb64ce427117"
server_key_iv: "1c7ebc53ac1db38d92b6447c0bbcebf7"
server_key_version: "encryptionkey@0"
I20260928 21:49:15.150287 6187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:15.158671 6187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:15.160323 6187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:15.160446 6187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:15.160691 6187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:15.160962 6187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:15.160995 6187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:15.161021 6187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:15.161044 6187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:15.166339 6187 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:42147
I20260928 21:49:15.166394 6315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:42147 every 8 connection(s)
I20260928 21:49:15.166664 6187 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:49:15.170084 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 6187
I20260928 21:49:15.170158 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:15.170312 3075 external_mini_cluster.cc:1428] Setting key 8d03d564b1a0eec9b58be14ee4685b3d
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632155, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:15.176033 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:15.167297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:35281 (local address 127.3.0.254:33651)
0928 21:49:15.167348 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:15.167350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:15.167534 (+ 184us) server_negotiation.cc:407] Connection header received
0928 21:49:15.168043 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:15.168045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:15.168060 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:15.168111 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:15.169094 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.169600 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.170745 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.170995 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.171070 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:15.173224 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:15.173225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:15.173226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:15.173232 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:15.174508 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:15.174995 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:15.174995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:15.174995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:15.175043 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:15.175291 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:15.175291 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:15.175291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:15.175387 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:15.175439 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:15.175816 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 21:49:15.175911 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:49:15.176168 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:15.167259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33651 (local address 127.3.0.193:35281)
0928 21:49:15.167422 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:15.167456 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:15.167948 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:15.168181 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:15.168185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:15.168808 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:15.169000 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:15.169005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.169684 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.169685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:15.170649 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:15.170652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.171084 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.171085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:15.171184 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:15.171758 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:15.171767 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:15.173142 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:15.174616 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:15.174623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:15.174626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:15.174920 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:15.175099 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:15.175099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:15.175100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:15.175234 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:15.175442 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:15.175449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:15.175620 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 21:49:15.175747 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 21:49:15.175927 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260928 21:49:15.176801 6316 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33651
I20260928 21:49:15.176908 6316 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:15.177101 6316 heartbeater.cc:511] Master 127.3.0.254:33651 requested a full tablet report, sending...
I20260928 21:49:15.177619 6125 ts_manager.cc:194] Registered new tserver with Master: 4fa81a9cf2a346e48c46de6aadc360a3 (127.3.0.193:42147)
I20260928 21:49:15.178201 6125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:35281
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:49:15.195072 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33651
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:15.273811 6321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:15.274025 6321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:15.274048 6321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:15.275154 6321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:15.275192 6321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:15.275215 6321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:15.276641 6321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:15.276827 6321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:15.277005 6321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:15.281795 6329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:15.281802 6326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:15.281802 6327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:15.282032 6321 server_base.cc:1034] running on GCE node
I20260928 21:49:15.282226 6321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:15.282433 6321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:15.283582 6321 hybrid_clock.cc:648] HybridClock initialized: now 1790632155283571 us; error 32 us; skew 500 ppm
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632155, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:15.286217 6321 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:49:15.286651 6321 webserver.cc:460] Webserver started at http://127.3.0.194:46241/ using document root <none> and password file <none>
I20260928 21:49:15.286857 6321 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:15.286903 6321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:15.287027 6321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:15.288031 6321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "bc0f4b75875844389bf8bf532fbeb9ed"
format_stamp: "Formatted at 2026-09-28 21:49:15 on dist-test-slave-jrck"
server_key: "d7da0310087b75eb92e4559092527913"
server_key_iv: "f71162243b3bc08a07d0efd995a0d02f"
server_key_version: "encryptionkey@0"
I20260928 21:49:15.288355 6321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "bc0f4b75875844389bf8bf532fbeb9ed"
format_stamp: "Formatted at 2026-09-28 21:49:15 on dist-test-slave-jrck"
server_key: "d7da0310087b75eb92e4559092527913"
server_key_iv: "f71162243b3bc08a07d0efd995a0d02f"
server_key_version: "encryptionkey@0"
I20260928 21:49:15.289630 6321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:15.290266 6336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:15.290421 6321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:15.290499 6321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "bc0f4b75875844389bf8bf532fbeb9ed"
format_stamp: "Formatted at 2026-09-28 21:49:15 on dist-test-slave-jrck"
server_key: "d7da0310087b75eb92e4559092527913"
server_key_iv: "f71162243b3bc08a07d0efd995a0d02f"
server_key_version: "encryptionkey@0"
I20260928 21:49:15.290563 6321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:15.303133 6321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:15.305166 6321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:15.305301 6321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:15.305517 6321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:15.305802 6321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:15.305837 6321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:15.305863 6321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:15.305886 6321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:15.311090 6321 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:36119
I20260928 21:49:15.311153 6449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:36119 every 8 connection(s)
I20260928 21:49:15.311429 6321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632155, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:15.319516 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 6321
I20260928 21:49:15.319618 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:15.319832 3075 external_mini_cluster.cc:1428] Setting key fdf0293a22515fc1b8ce7fbab8785339
I20260928 21:49:15.319942 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:15.312060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:46363 (local address 127.3.0.254:33651)
0928 21:49:15.312104 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:15.312105 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:15.312288 (+ 183us) server_negotiation.cc:407] Connection header received
0928 21:49:15.312779 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:15.312780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:15.312790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:15.312845 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:15.313706 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.314217 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.315292 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.315482 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.315533 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:15.317514 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:15.317516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:15.317516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:15.317522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:15.318698 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:15.319068 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:15.319069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:15.319069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:15.319121 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:15.319308 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:15.319309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:15.319309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:15.319424 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:15.319472 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:15.319787 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 21:49:15.319855 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:49:15.320276 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:15.312026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33651 (local address 127.3.0.194:46363)
0928 21:49:15.312170 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:15.312210 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:15.312699 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:15.312898 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:15.312901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:15.313453 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:15.313636 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:15.313641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.314298 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.314298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:15.315228 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:15.315230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.315540 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.315540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:15.315639 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:15.316098 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:15.316106 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:15.317411 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:15.318776 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:15.318782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:15.318784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:15.319002 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:15.319170 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:15.319171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:15.319171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:15.319262 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:15.319479 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:15.319486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:15.319656 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 21:49:15.319771 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 21:49:15.319946 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260928 21:49:15.320775 6450 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33651
I20260928 21:49:15.320876 6450 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:15.321045 6450 heartbeater.cc:511] Master 127.3.0.254:33651 requested a full tablet report, sending...
I20260928 21:49:15.321379 6125 ts_manager.cc:194] Registered new tserver with Master: bc0f4b75875844389bf8bf532fbeb9ed (127.3.0.194:36119)
I20260928 21:49:15.321738 6125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:46363
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:49:15.342942 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33651
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:15.423465 6455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:15.423676 6455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:15.423698 6455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:15.424887 6455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:15.424930 6455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:15.424959 6455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:15.426389 6455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:15.426615 6455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:15.426824 6455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:15.431545 6463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:15.431520 6460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:15.431612 6455 server_base.cc:1034] running on GCE node
W20260928 21:49:15.431525 6461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:15.431936 6455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:15.432150 6455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:15.433283 6455 hybrid_clock.cc:648] HybridClock initialized: now 1790632155433270 us; error 27 us; skew 500 ppm
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632155, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:15.435664 6455 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:49:15.436117 6455 webserver.cc:460] Webserver started at http://127.3.0.195:42711/ using document root <none> and password file <none>
I20260928 21:49:15.436326 6455 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:15.436373 6455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:15.436532 6455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:15.437562 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "5253b82a8314435d98cd87e62c209a9f"
format_stamp: "Formatted at 2026-09-28 21:49:15 on dist-test-slave-jrck"
server_key: "3599da1ae520290ed66aaea692749209"
server_key_iv: "05cf803da8029b03f6bbea8415e461d7"
server_key_version: "encryptionkey@0"
I20260928 21:49:15.437882 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "5253b82a8314435d98cd87e62c209a9f"
format_stamp: "Formatted at 2026-09-28 21:49:15 on dist-test-slave-jrck"
server_key: "3599da1ae520290ed66aaea692749209"
server_key_iv: "05cf803da8029b03f6bbea8415e461d7"
server_key_version: "encryptionkey@0"
I20260928 21:49:15.439085 6455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:49:15.439733 6470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:15.439879 6455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:15.439958 6455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "5253b82a8314435d98cd87e62c209a9f"
format_stamp: "Formatted at 2026-09-28 21:49:15 on dist-test-slave-jrck"
server_key: "3599da1ae520290ed66aaea692749209"
server_key_iv: "05cf803da8029b03f6bbea8415e461d7"
server_key_version: "encryptionkey@0"
I20260928 21:49:15.440019 6455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:15.450809 6455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:15.452921 6455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:15.453094 6455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:15.453306 6455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:15.453608 6455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:15.453644 6455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:15.453671 6455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:15.453696 6455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:15.459179 6455 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:39361
I20260928 21:49:15.459218 6583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:39361 every 8 connection(s)
I20260928 21:49:15.459542 6455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632155, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:15.467564 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 6455
I20260928 21:49:15.467664 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:15.467849 3075 external_mini_cluster.cc:1428] Setting key 1fb3f030cf0a0324fc40848cb85eb823
I20260928 21:49:15.468323 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:15.460188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:51505 (local address 127.3.0.254:33651)
0928 21:49:15.460283 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:15.460285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:15.460441 (+ 156us) server_negotiation.cc:407] Connection header received
0928 21:49:15.460959 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:15.460960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:15.460971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:15.461033 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:15.461921 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.462433 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.463385 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.463584 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.463641 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:15.465867 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:15.465868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:15.465868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:15.465876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:15.467010 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:15.467372 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:15.467374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:15.467374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:15.467429 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:15.467644 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:15.467645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:15.467646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:15.467772 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:15.467830 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:15.468167 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 21:49:15.468234 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:49:15.468528 6586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:15.460154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33651 (local address 127.3.0.195:51505)
0928 21:49:15.460324 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:15.460366 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:15.460873 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:15.461100 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:15.461104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:15.461645 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:15.461831 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:15.461835 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.462501 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.462502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:15.463303 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:15.463306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.463684 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.463685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:15.463816 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:15.464342 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:15.464350 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:15.465780 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:15.467094 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:15.467101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:15.467103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:15.467304 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:15.467483 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:15.467484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:15.467485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:15.467589 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:15.467827 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:15.467834 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:15.467983 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 21:49:15.468076 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:49:15.468239 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260928 21:49:15.468892 6584 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33651
I20260928 21:49:15.468977 6584 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:15.469115 6584 heartbeater.cc:511] Master 127.3.0.254:33651 requested a full tablet report, sending...
I20260928 21:49:15.469440 6125 ts_manager.cc:194] Registered new tserver with Master: 5253b82a8314435d98cd87e62c209a9f (127.3.0.195:39361)
I20260928 21:49:15.469765 6125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:51505
I20260928 21:49:15.479769 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632155, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632155, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:15.500798 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:15.494463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.3.0.254:33651)
0928 21:49:15.494531 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:15.494533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:15.494579 (+ 46us) server_negotiation.cc:407] Connection header received
0928 21:49:15.494660 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:15.494661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:15.494671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:15.494714 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:15.495240 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.495721 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.496442 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:15.496660 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:15.496716 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:15.498721 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:15.498722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:15.498722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:15.498729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:15.499749 (+ 1020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:15.500038 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:15.500040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:15.500040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:15.500090 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:15.500259 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:15.500259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:15.500260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:15.500407 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:15.500515 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:15.500634 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 21:49:15.500707 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 21:49:15.501012 6125 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54266
I20260928 21:49:15.501363 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 6187
I20260928 21:49:15.505678 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 6321
I20260928 21:49:15.509181 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 6455
I20260928 21:49:15.512513 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 6093
2026-09-28T21:49:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1674 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6596](info): set up 2 sockets
Sep 28 21:49:15 dist-test-slave-jrck krb5kdc[6596](info): commencing operation
krb5kdc: starting...
W20260928 21:49:16.456018 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:16 dist-test-slave-jrck krb5kdc[6596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:16.560137 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:45087
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:16.637028 6609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:16.637223 6609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:16.637253 6609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:16.638377 6609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:16.638418 6609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:16.638444 6609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:16.638460 6609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:16.638476 6609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:16.639962 6609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:45087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:16.640194 6609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:16.640405 6609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:16.644835 6614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:16.644840 6615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:16.644873 6617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:16.644876 6609 server_base.cc:1034] running on GCE node
I20260928 21:49:16.645232 6609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:16.645464 6609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:16.646597 6609 hybrid_clock.cc:648] HybridClock initialized: now 1790632156646583 us; error 31 us; skew 500 ppm
Sep 28 21:49:16 dist-test-slave-jrck krb5kdc[6596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:16.652779 6609 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:49:16.653203 6609 webserver.cc:460] Webserver started at http://127.3.0.254:45493/ using document root <none> and password file <none>
I20260928 21:49:16.653414 6609 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:16.653468 6609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:16.653589 6609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:16.654616 6609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "ae9ee64154614424ba0349620be9280e"
format_stamp: "Formatted at 2026-09-28 21:49:16 on dist-test-slave-jrck"
server_key: "a0bf55e774ab3d9917d0eb0d1440cc3d"
server_key_iv: "9fd3a296adf2b17a9afa948dff61561b"
server_key_version: "encryptionkey@0"
I20260928 21:49:16.654927 6609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "ae9ee64154614424ba0349620be9280e"
format_stamp: "Formatted at 2026-09-28 21:49:16 on dist-test-slave-jrck"
server_key: "a0bf55e774ab3d9917d0eb0d1440cc3d"
server_key_iv: "9fd3a296adf2b17a9afa948dff61561b"
server_key_version: "encryptionkey@0"
I20260928 21:49:16.656086 6609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:49:16.656735 6624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:16.656878 6609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:16.656958 6609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "ae9ee64154614424ba0349620be9280e"
format_stamp: "Formatted at 2026-09-28 21:49:16 on dist-test-slave-jrck"
server_key: "a0bf55e774ab3d9917d0eb0d1440cc3d"
server_key_iv: "9fd3a296adf2b17a9afa948dff61561b"
server_key_version: "encryptionkey@0"
I20260928 21:49:16.657018 6609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:16.667475 6609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:16.669397 6609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:16.669526 6609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:16.673161 6609 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:45087
I20260928 21:49:16.673193 6676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:45087 every 8 connection(s)
I20260928 21:49:16.673636 6609 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:16.674039 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 6609
I20260928 21:49:16.674139 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:16.674325 3075 external_mini_cluster.cc:1428] Setting key 8a957fcd5e8117b33dfac1273e6ae617
I20260928 21:49:16.674324 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:16.676983 6677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e: Bootstrap starting.
I20260928 21:49:16.677536 6677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:16.677771 6677 log.cc:824] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:16.678467 6677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e: No bootstrap required, opened a new log
I20260928 21:49:16.680439 6677 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9ee64154614424ba0349620be9280e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45087 } }
I20260928 21:49:16.680631 6677 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 28 21:49:16 dist-test-slave-jrck krb5kdc[6596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:16.680672 6677 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9ee64154614424ba0349620be9280e, State: Initialized, Role: FOLLOWER
I20260928 21:49:16.680794 6677 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9ee64154614424ba0349620be9280e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45087 } }
I20260928 21:49:16.680869 6677 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:16.680899 6677 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:16.680946 6677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:16.681569 6677 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9ee64154614424ba0349620be9280e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45087 } }
I20260928 21:49:16.681681 6677 leader_election.cc:302] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae9ee64154614424ba0349620be9280e; no voters:
I20260928 21:49:16.681835 6677 leader_election.cc:288] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:16.681874 6682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:16.682044 6682 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [term 1 LEADER]: Becoming Leader. State: Replica: ae9ee64154614424ba0349620be9280e, State: Running, Role: LEADER
I20260928 21:49:16.682075 6677 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:16.682193 6682 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9ee64154614424ba0349620be9280e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45087 } }
I20260928 21:49:16.682574 6684 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae9ee64154614424ba0349620be9280e. Latest consensus state: current_term: 1 leader_uuid: "ae9ee64154614424ba0349620be9280e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9ee64154614424ba0349620be9280e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45087 } } }
I20260928 21:49:16.682653 6684 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:16.682875 6683 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae9ee64154614424ba0349620be9280e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae9ee64154614424ba0349620be9280e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45087 } } }
I20260928 21:49:16.683197 6690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:16.683200 6683 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:16.683746 6690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:16.684684 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:16.674787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.3.0.254:45087)
0928 21:49:16.675005 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:16.675012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:16.675035 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:49:16.675589 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:16.675596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:16.675641 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:16.675762 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:16.676337 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:16.677308 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:16.678177 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:16.678754 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:16.678894 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:16.681115 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:16.681119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:16.681120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:16.681130 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:16.682884 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:16.683355 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:16.683358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:16.683359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:16.683453 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:16.683668 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:16.683669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:16.683670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:16.683934 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:16.684071 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:16.684378 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 21:49:16.684532 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260928 21:49:16.685228 6690 catalog_manager.cc:1347] Generated new cluster ID: c87e02dafb104f929a75bf1153ba90cb
I20260928 21:49:16.685272 6690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:49:16.700037 6690 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:16.700580 6690 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:16.708019 6690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae9ee64154614424ba0349620be9280e: Generated new TSK 0
I20260928 21:49:16.708168 6690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:49:16.736588 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45087
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:16.814237 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:16.814424 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:16.814445 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:16.815550 6703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:16.815591 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:16.815616 6703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:16.817107 6703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:16.817318 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:16.817519 6703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:16.821995 6709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:16.822021 6711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:16.822034 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:16.822132 6703 server_base.cc:1034] running on GCE node
I20260928 21:49:16.822377 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:16.822573 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:16.823709 6703 hybrid_clock.cc:648] HybridClock initialized: now 1790632156823698 us; error 30 us; skew 500 ppm
Sep 28 21:49:16 dist-test-slave-jrck krb5kdc[6596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:16.826061 6703 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:49:16.826462 6703 webserver.cc:460] Webserver started at http://127.3.0.193:36045/ using document root <none> and password file <none>
I20260928 21:49:16.826661 6703 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:16.826702 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:16.826817 6703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:16.827752 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "7e36d42f7d144ece9fd170fd147140bc"
format_stamp: "Formatted at 2026-09-28 21:49:16 on dist-test-slave-jrck"
server_key: "157b9ccea6984975476424740b4ee885"
server_key_iv: "9529726905657da8e12aa6e6371bab37"
server_key_version: "encryptionkey@0"
I20260928 21:49:16.828055 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "7e36d42f7d144ece9fd170fd147140bc"
format_stamp: "Formatted at 2026-09-28 21:49:16 on dist-test-slave-jrck"
server_key: "157b9ccea6984975476424740b4ee885"
server_key_iv: "9529726905657da8e12aa6e6371bab37"
server_key_version: "encryptionkey@0"
I20260928 21:49:16.829250 6703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:16.829854 6718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:16.830017 6703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:16.830089 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "7e36d42f7d144ece9fd170fd147140bc"
format_stamp: "Formatted at 2026-09-28 21:49:16 on dist-test-slave-jrck"
server_key: "157b9ccea6984975476424740b4ee885"
server_key_iv: "9529726905657da8e12aa6e6371bab37"
server_key_version: "encryptionkey@0"
I20260928 21:49:16.830152 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:16.858784 6703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:16.861006 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:16.861135 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:16.861341 6703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:16.861640 6703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:16.861672 6703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:16.861694 6703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:16.861716 6703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:16.867098 6703 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:41163
I20260928 21:49:16.867146 6831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:41163 every 8 connection(s)
I20260928 21:49:16.867439 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:49:16.870977 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 6703
I20260928 21:49:16.871074 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:16.871263 3075 external_mini_cluster.cc:1428] Setting key 3f51b6e48cb2635f6d4e0e5e2164c2af
Sep 28 21:49:16 dist-test-slave-jrck krb5kdc[6596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:16.877049 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:16.868081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:52441 (local address 127.3.0.254:45087)
0928 21:49:16.868134 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:16.868136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:16.868324 (+ 188us) server_negotiation.cc:407] Connection header received
0928 21:49:16.869298 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:16.869300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:16.869312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:16.869358 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:16.869963 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:16.870473 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:16.871696 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:16.871967 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:16.872022 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:16.874457 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:16.874459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:16.874459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:16.874466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:16.875770 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:16.876149 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:16.876150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:16.876151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:16.876210 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:16.876419 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:16.876420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:16.876420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:16.876562 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:16.876617 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:16.876875 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 21:49:16.876958 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:49:16.877209 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:16.868064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:45087 (local address 127.3.0.193:52441)
0928 21:49:16.868212 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:16.868246 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:16.869196 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:16.869439 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:16.869443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:16.869685 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:16.869867 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:16.869871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:16.870555 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:16.870556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:16.871619 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:16.871622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:16.872039 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:16.872040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:16.872193 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:16.872697 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:16.872706 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:16.874376 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:16.875851 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:16.875858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:16.875861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:16.876081 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:16.876261 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:16.876261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:16.876262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:16.876364 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:16.876616 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:16.876623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:16.876751 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 21:49:16.876846 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:49:16.876999 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260928 21:49:16.877635 6832 heartbeater.cc:347] Connected to a master server at 127.3.0.254:45087
I20260928 21:49:16.877740 6832 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:16.877948 6832 heartbeater.cc:511] Master 127.3.0.254:45087 requested a full tablet report, sending...
I20260928 21:49:16.878468 6641 ts_manager.cc:194] Registered new tserver with Master: 7e36d42f7d144ece9fd170fd147140bc (127.3.0.193:41163)
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
I20260928 21:49:16.879093 6641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:52441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:49:16.893105 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45087
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:16.969913 6837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:16.970111 6837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:16.970134 6837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:16.971278 6837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:16.971318 6837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:16.971345 6837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:16.972852 6837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:16.973069 6837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:16.973265 6837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:16.977924 6845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:16.978037 6837 server_base.cc:1034] running on GCE node
W20260928 21:49:16.977896 6842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:16.977900 6843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:16.978318 6837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:16.978539 6837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:16.979681 6837 hybrid_clock.cc:648] HybridClock initialized: now 1790632156979668 us; error 25 us; skew 500 ppm
Sep 28 21:49:16 dist-test-slave-jrck krb5kdc[6596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:16.982015 6837 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:49:16.982501 6837 webserver.cc:460] Webserver started at http://127.3.0.194:46275/ using document root <none> and password file <none>
I20260928 21:49:16.982703 6837 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:16.982749 6837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:16.982874 6837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:16.983919 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "030c2a81fd56407abedfe37a982fa288"
format_stamp: "Formatted at 2026-09-28 21:49:16 on dist-test-slave-jrck"
server_key: "5c9dc759b985f6b1faed15c7bece97d6"
server_key_iv: "36528608f2d117a1d709d68520a81639"
server_key_version: "encryptionkey@0"
I20260928 21:49:16.984236 6837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "030c2a81fd56407abedfe37a982fa288"
format_stamp: "Formatted at 2026-09-28 21:49:16 on dist-test-slave-jrck"
server_key: "5c9dc759b985f6b1faed15c7bece97d6"
server_key_iv: "36528608f2d117a1d709d68520a81639"
server_key_version: "encryptionkey@0"
I20260928 21:49:16.985458 6837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:16.986136 6852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:16.986289 6837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:16.986382 6837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "030c2a81fd56407abedfe37a982fa288"
format_stamp: "Formatted at 2026-09-28 21:49:16 on dist-test-slave-jrck"
server_key: "5c9dc759b985f6b1faed15c7bece97d6"
server_key_iv: "36528608f2d117a1d709d68520a81639"
server_key_version: "encryptionkey@0"
I20260928 21:49:16.986443 6837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:17.002194 6837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:17.004212 6837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:17.004343 6837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:17.004557 6837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:17.004832 6837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:17.004868 6837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:17.004895 6837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:17.004913 6837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:17.010255 6837 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:34741
I20260928 21:49:17.010310 6965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:34741 every 8 connection(s)
I20260928 21:49:17.010592 6837 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:49:17 dist-test-slave-jrck krb5kdc[6596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:17.017117 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 6837
I20260928 21:49:17.017204 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:17.017351 3075 external_mini_cluster.cc:1428] Setting key 76b7ed7393afdc9bd0c73fed94e4bdfc
I20260928 21:49:17.019129 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:17.011237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:49065 (local address 127.3.0.254:45087)
0928 21:49:17.011292 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:17.011294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:17.011492 (+ 198us) server_negotiation.cc:407] Connection header received
0928 21:49:17.012373 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:17.012374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:17.012391 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:17.012444 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:17.013023 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.013516 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.014462 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.014634 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.014677 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:17.016544 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:17.016546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:17.016547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:17.016553 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:17.017844 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:17.018274 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:17.018274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:17.018275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:17.018323 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:17.018577 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:17.018577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:17.018578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:17.018670 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:17.018719 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:17.019002 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 21:49:17.019069 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:49:17.019336 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:17.011214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:45087 (local address 127.3.0.194:49065)
0928 21:49:17.011374 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:17.011415 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:17.012285 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:17.012525 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:17.012529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:17.012771 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:17.012960 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:17.012965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.013588 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.013589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:17.014401 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:17.014403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.014685 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.014686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:17.014775 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:17.015158 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:17.015165 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:17.016444 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:17.017925 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:17.017931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:17.017933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:17.018211 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:17.018396 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:17.018397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:17.018397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:17.018524 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:17.018716 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:17.018721 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:17.018868 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 21:49:17.018979 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:49:17.019122 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260928 21:49:17.019870 6966 heartbeater.cc:347] Connected to a master server at 127.3.0.254:45087
I20260928 21:49:17.019959 6966 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:17.020113 6966 heartbeater.cc:511] Master 127.3.0.254:45087 requested a full tablet report, sending...
I20260928 21:49:17.020476 6641 ts_manager.cc:194] Registered new tserver with Master: 030c2a81fd56407abedfe37a982fa288 (127.3.0.194:34741)
I20260928 21:49:17.020941 6641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:49065
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:49:17.039373 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45087
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:17.116328 6971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:17.116573 6971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:17.116601 6971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:17.117759 6971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:17.117798 6971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:17.117848 6971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:17.119340 6971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:17.119557 6971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:17.119755 6971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:17.124342 6979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:17.124402 6971 server_base.cc:1034] running on GCE node
W20260928 21:49:17.124328 6977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:17.124331 6976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:17.124773 6971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:17.125003 6971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:17.126147 6971 hybrid_clock.cc:648] HybridClock initialized: now 1790632157126129 us; error 28 us; skew 500 ppm
Sep 28 21:49:17 dist-test-slave-jrck krb5kdc[6596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632157, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:17.128588 6971 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:49:17.128990 6971 webserver.cc:460] Webserver started at http://127.3.0.195:34085/ using document root <none> and password file <none>
I20260928 21:49:17.129170 6971 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:17.129208 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:17.129298 6971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:17.130227 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "295a6647d4f44bed99c66385837fffad"
format_stamp: "Formatted at 2026-09-28 21:49:17 on dist-test-slave-jrck"
server_key: "88a6fd78ae07441d8304773b04359900"
server_key_iv: "b40328d8ecc7b1d8ed53f8b9c3096bbb"
server_key_version: "encryptionkey@0"
I20260928 21:49:17.130566 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "295a6647d4f44bed99c66385837fffad"
format_stamp: "Formatted at 2026-09-28 21:49:17 on dist-test-slave-jrck"
server_key: "88a6fd78ae07441d8304773b04359900"
server_key_iv: "b40328d8ecc7b1d8ed53f8b9c3096bbb"
server_key_version: "encryptionkey@0"
I20260928 21:49:17.131671 6971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:17.132243 6986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:17.132385 6971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:17.132476 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "295a6647d4f44bed99c66385837fffad"
format_stamp: "Formatted at 2026-09-28 21:49:17 on dist-test-slave-jrck"
server_key: "88a6fd78ae07441d8304773b04359900"
server_key_iv: "b40328d8ecc7b1d8ed53f8b9c3096bbb"
server_key_version: "encryptionkey@0"
I20260928 21:49:17.132539 6971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:17.143208 6971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:17.144830 6971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:17.144943 6971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:17.145117 6971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:17.145355 6971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:17.145383 6971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:17.145403 6971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:17.145419 6971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:17.150566 6971 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:44449
I20260928 21:49:17.150605 7099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:44449 every 8 connection(s)
I20260928 21:49:17.150889 6971 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:49:17.153348 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 6971
I20260928 21:49:17.153437 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:17.153602 3075 external_mini_cluster.cc:1428] Setting key a28cd752842d6e37a92e5d112e1fb32a
Sep 28 21:49:17 dist-test-slave-jrck krb5kdc[6596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632157, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:17.159871 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:17.151607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:48689 (local address 127.3.0.254:45087)
0928 21:49:17.151671 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:17.151673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:17.151876 (+ 203us) server_negotiation.cc:407] Connection header received
0928 21:49:17.152794 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:17.152795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:17.152805 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:17.152859 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:17.153443 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.154143 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.155211 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.155440 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.155503 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:17.157410 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:17.157412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:17.157412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:17.157419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:17.158667 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:17.159019 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:17.159020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:17.159021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:17.159078 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:17.159262 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:17.159262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:17.159263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:17.159356 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:17.159411 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:17.159694 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 21:49:17.159778 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:49:17.160077 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:17.151565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:45087 (local address 127.3.0.195:48689)
0928 21:49:17.151736 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:17.151795 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:17.152711 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:17.152916 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:17.152919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:17.153158 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:17.153343 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:17.153348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.154229 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.154230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:17.155122 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:17.155125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.155503 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.155503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:17.155599 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:17.156023 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:17.156030 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:17.157321 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:17.158731 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:17.158735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:17.158737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:17.158967 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:17.159134 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:17.159135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:17.159135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:17.159225 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:17.159430 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:17.159436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:17.159572 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 21:49:17.159663 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:49:17.159845 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260928 21:49:17.160516 7100 heartbeater.cc:347] Connected to a master server at 127.3.0.254:45087
I20260928 21:49:17.160604 7100 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:17.160745 7100 heartbeater.cc:511] Master 127.3.0.254:45087 requested a full tablet report, sending...
I20260928 21:49:17.161118 6641 ts_manager.cc:194] Registered new tserver with Master: 295a6647d4f44bed99c66385837fffad (127.3.0.195:44449)
I20260928 21:49:17.161446 6641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:48689
I20260928 21:49:17.165087 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:49:17 dist-test-slave-jrck krb5kdc[6596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:49:17 dist-test-slave-jrck krb5kdc[6596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:49:17.186023 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:17.178867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.3.0.193:41163)
0928 21:49:17.179018 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:17.179023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:17.179033 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:49:17.179058 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:17.179060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:17.179073 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:17.179141 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:17.179612 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.180329 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.181047 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.181406 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.181461 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:17.183611 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:17.183623 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:17.183623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:17.183630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:17.185087 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:17.185377 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:17.185380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:17.185380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:17.185442 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:17.185574 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:17.185574 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:17.185574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:17.185691 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:17.185785 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:17.185838 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 21:49:17.185912 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
Sep 28 21:49:17 dist-test-slave-jrck krb5kdc[6596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:17.193199 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:17.187208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.3.0.254:45087)
0928 21:49:17.187281 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:17.187283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:17.187294 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:49:17.187318 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:17.187319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:17.187328 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:17.187387 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:17.187819 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.188343 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.189120 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:17.189303 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:17.189347 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:17.191241 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:17.191242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:17.191242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:17.191249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:17.192381 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:17.192653 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:17.192653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:17.192654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:17.192698 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:17.192817 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:17.192818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:17.192818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:17.192950 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:17.193019 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:17.193083 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 21:49:17.193137 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 21:49:17.193391 6641 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57398
I20260928 21:49:17.193681 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 6703
I20260928 21:49:17.197515 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 6837
I20260928 21:49:17.201143 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 6971
I20260928 21:49:17.204806 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 6609
2026-09-28T21:49:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1692 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:17 dist-test-slave-jrck krb5kdc[7112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:17 dist-test-slave-jrck krb5kdc[7112](info): set up 2 sockets
Sep 28 21:49:17 dist-test-slave-jrck krb5kdc[7112](info): commencing operation
krb5kdc: starting...
W20260928 21:49:18.152017 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:18 dist-test-slave-jrck krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:18.256909 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:43401
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:43401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:18.334745 7125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:18.334947 7125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:18.334968 7125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:18.336107 7125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:18.336146 7125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:18.336159 7125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:18.336172 7125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:18.336195 7125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:18.337697 7125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:43401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:43401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:18.337916 7125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:18.338124 7125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:18.342576 7130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:18.342592 7131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:18.342578 7133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:18.342592 7125 server_base.cc:1034] running on GCE node
I20260928 21:49:18.342973 7125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:18.343254 7125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:18.344395 7125 hybrid_clock.cc:648] HybridClock initialized: now 1790632158344381 us; error 31 us; skew 500 ppm
Sep 28 21:49:18 dist-test-slave-jrck krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:18.346697 7125 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:49:18.347010 7125 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:18.347060 7125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:18.347185 7125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:18.348199 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "adf4511c76324cbb8f926de6e678b9d9"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "63ceb017bc46cab86663e4c5e78ed877"
server_key_iv: "9b54f15fd2600d770d528992afbe8727"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.348567 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "adf4511c76324cbb8f926de6e678b9d9"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "63ceb017bc46cab86663e4c5e78ed877"
server_key_iv: "9b54f15fd2600d770d528992afbe8727"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.349702 7125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:18.350337 7139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:18.350482 7125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:18.350560 7125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "adf4511c76324cbb8f926de6e678b9d9"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "63ceb017bc46cab86663e4c5e78ed877"
server_key_iv: "9b54f15fd2600d770d528992afbe8727"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.350633 7125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:18.384517 7125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:18.386340 7125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:18.386451 7125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:18.390271 7125 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:43401
I20260928 21:49:18.390295 7191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:43401 every 8 connection(s)
I20260928 21:49:18.390611 7125 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:18.391185 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 7125
I20260928 21:49:18.391268 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:18.391434 3075 external_mini_cluster.cc:1428] Setting key 49e49a3d966ce0924c49ceefcda4f25d
I20260928 21:49:18.391510 7192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:18.394927 7192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9: Bootstrap starting.
I20260928 21:49:18.395553 7192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:18.395830 7192 log.cc:824] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:18.396502 7192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9: No bootstrap required, opened a new log
Sep 28 21:49:18 dist-test-slave-jrck krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:18.398694 7192 raft_consensus.cc:348] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4511c76324cbb8f926de6e678b9d9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43401 } }
I20260928 21:49:18.398864 7192 raft_consensus.cc:374] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:18.398910 7192 raft_consensus.cc:729] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf4511c76324cbb8f926de6e678b9d9, State: Initialized, Role: FOLLOWER
I20260928 21:49:18.399058 7192 consensus_queue.cc:260] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4511c76324cbb8f926de6e678b9d9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43401 } }
I20260928 21:49:18.399145 7192 raft_consensus.cc:388] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:18.399176 7192 raft_consensus.cc:482] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:18.399226 7192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:18.399868 7192 raft_consensus.cc:504] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4511c76324cbb8f926de6e678b9d9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43401 } }
I20260928 21:49:18.399972 7192 leader_election.cc:302] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adf4511c76324cbb8f926de6e678b9d9; no voters:
I20260928 21:49:18.400134 7192 leader_election.cc:288] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:18.400177 7197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:18.400359 7197 raft_consensus.cc:686] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [term 1 LEADER]: Becoming Leader. State: Replica: adf4511c76324cbb8f926de6e678b9d9, State: Running, Role: LEADER
I20260928 21:49:18.400410 7192 sys_catalog.cc:564] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:18.400491 7197 consensus_queue.cc:237] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4511c76324cbb8f926de6e678b9d9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43401 } }
I20260928 21:49:18.400792 7197 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adf4511c76324cbb8f926de6e678b9d9. Latest consensus state: current_term: 1 leader_uuid: "adf4511c76324cbb8f926de6e678b9d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4511c76324cbb8f926de6e678b9d9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43401 } } }
I20260928 21:49:18.401005 7197 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:18.400944 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adf4511c76324cbb8f926de6e678b9d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adf4511c76324cbb8f926de6e678b9d9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43401 } } }
I20260928 21:49:18.401371 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:18.401572 7206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:18.402022 7206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:18.403402 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.391874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.3.0.254:43401)
0928 21:49:18.392024 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.392034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.392067 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:49:18.392589 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.392599 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.392666 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.392820 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:18.393416 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.394732 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.395497 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.396105 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.396207 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.398551 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:18.398555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:18.398556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:18.398568 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:18.400407 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:18.400828 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:18.400832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:18.400832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:18.400941 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:18.401179 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:18.401180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:18.401180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:18.401492 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:18.402024 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.403061 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.403208 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260928 21:49:18.403972 7206 catalog_manager.cc:1347] Generated new cluster ID: 3496b163e6a24c1ebace76db8225e700
I20260928 21:49:18.404060 7206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:49:18.410260 7206 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:18.410755 7206 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:18.421887 7206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9: Generated new TSK 0
I20260928 21:49:18.422080 7206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:49:18.450742 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43401
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:18.542948 7218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:18.543140 7218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:18.543164 7218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:18.544279 7218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:18.544312 7218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:18.544338 7218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:18.545861 7218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:18.546044 7218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:18.546222 7218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:18.550805 7226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:18.550810 7223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:18.550812 7224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:18.551177 7218 server_base.cc:1034] running on GCE node
I20260928 21:49:18.551362 7218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:18.551573 7218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:18.552702 7218 hybrid_clock.cc:648] HybridClock initialized: now 1790632158552685 us; error 31 us; skew 500 ppm
Sep 28 21:49:18 dist-test-slave-jrck krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:18.555068 7218 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:49:18.555413 7218 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:18.555477 7218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:18.555604 7218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:18.556656 7218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "05ec5984f54e41b0ac4aecbde24cddcb"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "e4ec2d151cc2932cca9f2c565f48389b"
server_key_iv: "4f496907554e4f7e0467eacd70db69b9"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.556972 7218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "05ec5984f54e41b0ac4aecbde24cddcb"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "e4ec2d151cc2932cca9f2c565f48389b"
server_key_iv: "4f496907554e4f7e0467eacd70db69b9"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.558110 7218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:18.558744 7232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:18.558884 7218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:18.558956 7218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "05ec5984f54e41b0ac4aecbde24cddcb"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "e4ec2d151cc2932cca9f2c565f48389b"
server_key_iv: "4f496907554e4f7e0467eacd70db69b9"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.559020 7218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:18.577435 7218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:18.579202 7218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:18.579331 7218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:18.579566 7218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:18.579876 7218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:18.579909 7218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:18.579939 7218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:18.579957 7218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:18.585464 7218 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:35323
I20260928 21:49:18.585515 7345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:35323 every 8 connection(s)
I20260928 21:49:18.585764 7218 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:49:18 dist-test-slave-jrck krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:18.594128 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.586435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:43485 (local address 127.3.0.254:43401)
0928 21:49:18.586497 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.586499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.586673 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:49:18.587171 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.587172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.587182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.587237 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:18.588164 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.588728 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.589703 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.589901 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.589960 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.592007 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:18.592008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:18.592009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:18.592015 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:18.593125 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:18.593455 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:18.593456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:18.593456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:18.593499 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:18.593666 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:18.593666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:18.593667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:18.593758 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:18.593801 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.594001 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.594064 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:49:18.594287 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.586393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:43401 (local address 127.3.0.193:43485)
0928 21:49:18.586550 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:18.586592 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:18.587085 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:18.587302 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:18.587305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:18.587583 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:18.588084 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.588090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.588799 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.588800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.589627 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.589630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.589963 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.589963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.590060 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.590536 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:18.590544 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:18.591923 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:18.593188 (+ 1265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:18.593194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:18.593196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:18.593401 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:18.593540 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:18.593541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:18.593541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:18.593627 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:18.593798 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:18.593800 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:18.593911 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 21:49:18.593985 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:49:18.594100 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260928 21:49:18.594729 7346 heartbeater.cc:347] Connected to a master server at 127.3.0.254:43401
I20260928 21:49:18.594816 7346 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:18.594962 7346 heartbeater.cc:511] Master 127.3.0.254:43401 requested a full tablet report, sending...
I20260928 21:49:18.595431 7156 ts_manager.cc:194] Registered new tserver with Master: 05ec5984f54e41b0ac4aecbde24cddcb (127.3.0.193:35323)
I20260928 21:49:18.595688 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 7218
I20260928 21:49:18.595770 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:18.595937 3075 external_mini_cluster.cc:1428] Setting key cec6073f36e8b906e0b5067c756212b1
I20260928 21:49:18.596139 7156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:43485
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:49:18.616598 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43401
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:18.694269 7351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:18.694463 7351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:18.694485 7351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:18.695598 7351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:18.695637 7351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:18.695662 7351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:18.697134 7351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:18.697346 7351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:18.697551 7351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:18.702092 7359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:18.702107 7356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:18.702082 7357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:18.702219 7351 server_base.cc:1034] running on GCE node
I20260928 21:49:18.702507 7351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:18.702723 7351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:18.703853 7351 hybrid_clock.cc:648] HybridClock initialized: now 1790632158703841 us; error 28 us; skew 500 ppm
Sep 28 21:49:18 dist-test-slave-jrck krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:18.706063 7351 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:49:18.706354 7351 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:18.706421 7351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:18.706534 7351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:18.707494 7351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "9c5d21ebe0c5436fbd4edf85664387f3"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "1e5e03bf69aa8f6ec10936ca3ffca8ba"
server_key_iv: "782b527f4545686a3ea2b994b028c251"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.707798 7351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "9c5d21ebe0c5436fbd4edf85664387f3"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "1e5e03bf69aa8f6ec10936ca3ffca8ba"
server_key_iv: "782b527f4545686a3ea2b994b028c251"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.708945 7351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:18.709532 7365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:18.709674 7351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:18.709749 7351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "9c5d21ebe0c5436fbd4edf85664387f3"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "1e5e03bf69aa8f6ec10936ca3ffca8ba"
server_key_iv: "782b527f4545686a3ea2b994b028c251"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.709805 7351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:18.723913 7351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:18.725720 7351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:18.725829 7351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:18.726001 7351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:18.726229 7351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:18.726256 7351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:18.726275 7351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:18.726286 7351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:18.731518 7351 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:38363
I20260928 21:49:18.731567 7478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:38363 every 8 connection(s)
I20260928 21:49:18.731797 7351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:49:18 dist-test-slave-jrck krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:18.740116 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.732485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:59199 (local address 127.3.0.254:43401)
0928 21:49:18.732531 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.732533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.732725 (+ 192us) server_negotiation.cc:407] Connection header received
0928 21:49:18.733183 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.733183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.733192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.733245 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:18.734122 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.734651 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.735583 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.735759 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.735799 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.737873 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:18.737875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:18.737876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:18.737883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:18.738968 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:18.739353 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:18.739354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:18.739355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:18.739416 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:18.739626 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:18.739626 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:18.739627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:18.739730 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:18.739779 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.739985 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.740050 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:49:18.740337 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.732435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:43401 (local address 127.3.0.194:59199)
0928 21:49:18.732607 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:18.732649 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:18.733108 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:18.733319 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:18.733323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:18.733562 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:18.734046 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.734051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.734716 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.734716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.735521 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.735523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.735825 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.735826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.735920 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.736414 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:18.736422 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:18.737761 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:18.739042 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:18.739048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:18.739050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:18.739252 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:18.739476 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:18.739477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:18.739477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:18.739575 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:18.739774 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:18.739776 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:18.739889 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 21:49:18.739964 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:49:18.740081 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260928 21:49:18.740551 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 7351
I20260928 21:49:18.740650 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:18.740845 3075 external_mini_cluster.cc:1428] Setting key 347429954380a544eb231ce015d68290
I20260928 21:49:18.740833 7479 heartbeater.cc:347] Connected to a master server at 127.3.0.254:43401
I20260928 21:49:18.740916 7479 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:18.741066 7479 heartbeater.cc:511] Master 127.3.0.254:43401 requested a full tablet report, sending...
I20260928 21:49:18.741447 7155 ts_manager.cc:194] Registered new tserver with Master: 9c5d21ebe0c5436fbd4edf85664387f3 (127.3.0.194:38363)
I20260928 21:49:18.741827 7155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:59199
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:49:18.760923 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43401
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:18.838548 7484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:18.838734 7484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:18.838757 7484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:18.839870 7484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:18.839910 7484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:18.839938 7484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:18.841459 7484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:18.841675 7484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:18.841876 7484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:18.846329 7489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:18.846351 7490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:18.846328 7492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:18.846621 7484 server_base.cc:1034] running on GCE node
I20260928 21:49:18.846812 7484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:18.847024 7484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:18.848187 7484 hybrid_clock.cc:648] HybridClock initialized: now 1790632158848167 us; error 29 us; skew 500 ppm
Sep 28 21:49:18 dist-test-slave-jrck krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:18.850639 7484 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:49:18.850957 7484 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:18.851003 7484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:18.851118 7484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:18.852128 7484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "94d2c7957d3a4872a4e25fde9c58f2a6"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "5796040790868c52cbb8d42be6ac80f7"
server_key_iv: "f17e5b07fd2913cbb64edf9f4eee5083"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.852425 7484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "94d2c7957d3a4872a4e25fde9c58f2a6"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "5796040790868c52cbb8d42be6ac80f7"
server_key_iv: "f17e5b07fd2913cbb64edf9f4eee5083"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.853617 7484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:18.854301 7498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:18.854457 7484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:18.854537 7484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "94d2c7957d3a4872a4e25fde9c58f2a6"
format_stamp: "Formatted at 2026-09-28 21:49:18 on dist-test-slave-jrck"
server_key: "5796040790868c52cbb8d42be6ac80f7"
server_key_iv: "f17e5b07fd2913cbb64edf9f4eee5083"
server_key_version: "encryptionkey@0"
I20260928 21:49:18.854599 7484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:18.872812 7484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:18.874869 7484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:18.875013 7484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:18.875212 7484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:18.875499 7484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:18.875531 7484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:18.875560 7484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:18.875583 7484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:18.880678 7484 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:37511
I20260928 21:49:18.880731 7611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:37511 every 8 connection(s)
I20260928 21:49:18.880981 7484 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:49:18.884738 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 7484
I20260928 21:49:18.884821 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:18.885035 3075 external_mini_cluster.cc:1428] Setting key 7dbc2e2dbaaca678e192fe01cc86aadd
Sep 28 21:49:18 dist-test-slave-jrck krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:18.889536 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.881656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:38147 (local address 127.3.0.254:43401)
0928 21:49:18.881713 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.881714 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.881889 (+ 175us) server_negotiation.cc:407] Connection header received
0928 21:49:18.882378 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.882379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.882391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.882442 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:18.883358 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.883844 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.884927 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.885208 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.885278 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.887230 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:18.887232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:18.887232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:18.887239 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:18.888428 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:18.888786 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:18.888787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:18.888787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:18.888835 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:18.889007 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:18.889007 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:18.889007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:18.889100 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:18.889144 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.889399 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.889473 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:49:18.889621 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.881618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:43401 (local address 127.3.0.195:38147)
0928 21:49:18.881774 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:18.881809 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:18.882277 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:18.882537 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:18.882541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:18.882788 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:18.883257 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.883262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.883925 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.883926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.884805 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.884808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.885293 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.885294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.885436 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.885905 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:18.885915 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:18.887152 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:18.888541 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:18.888544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:18.888545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:18.888734 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:18.888876 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:18.888876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:18.888877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:18.888967 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:18.889140 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:18.889143 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:18.889252 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 21:49:18.889329 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:49:18.889443 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260928 21:49:18.890038 7612 heartbeater.cc:347] Connected to a master server at 127.3.0.254:43401
I20260928 21:49:18.890120 7612 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:18.890254 7612 heartbeater.cc:511] Master 127.3.0.254:43401 requested a full tablet report, sending...
I20260928 21:49:18.890584 7155 ts_manager.cc:194] Registered new tserver with Master: 94d2c7957d3a4872a4e25fde9c58f2a6 (127.3.0.195:37511)
I20260928 21:49:18.890923 7155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:38147
I20260928 21:49:18.896613 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:49:18.903446 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.897816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.3.0.254:43401)
0928 21:49:18.897859 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.897862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.897964 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:49:18.898025 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.898025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.898035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.898082 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:18.898623 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.899101 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.899798 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.899989 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.900042 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.901246 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:18.901247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:18.901248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:18.901255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:18.902437 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:18.902733 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:18.902734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:18.902735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:18.902786 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:18.902942 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:18.902943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:18.902943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:18.903084 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:18.903174 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.903286 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.903358 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:49:18.904742 7155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:18.906235 7155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:18.911922 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.908138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:38363 (local address 127.0.0.1:54100)
0928 21:49:18.908478 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:18.908486 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:18.908530 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:18.908683 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:18.908686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:18.908688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:18.908750 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.908752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.909663 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.909664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.910778 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.910780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.911637 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.911638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.911713 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.911729 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:49:18.911795 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:49:18.911848 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260928 21:49:18.911927 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.908241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.3.0.194:38363)
0928 21:49:18.908417 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.908422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.908523 (+ 101us) server_negotiation.cc:407] Connection header received
0928 21:49:18.908589 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.908591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.908605 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.908675 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:18.908801 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.909580 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.910869 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.911561 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.911627 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.911681 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.911815 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.911851 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20260928 21:49:18.912199 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.908435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.3.0.193:35323)
0928 21:49:18.908904 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.908908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.908916 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:18.908961 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.908962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.908977 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.909042 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:18.909231 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.910230 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.911194 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.911687 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.911748 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.911804 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.912091 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.912127 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":97,"threads_started":1}
I20260928 21:49:18.912031 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.908325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:35323 (local address 127.0.0.1:44184)
0928 21:49:18.908612 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:18.908622 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:18.908672 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:18.909119 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:18.909120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:18.909122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:18.909162 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.909164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.910308 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.910309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.911125 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.911127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.911833 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.911834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.911885 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.911888 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:18.911928 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 21:49:18.911972 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":32,"threads_started":1}
I20260928 21:49:18.912320 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.908421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.3.0.195:37511)
0928 21:49:18.908634 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.908640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.908649 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:49:18.908687 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.908689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.908703 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.908772 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:18.908961 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.910014 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.911235 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.911897 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.911958 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.912011 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.912213 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.912254 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":113,"threads_started":1}
I20260928 21:49:18.912272 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.908137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:37511 (local address 127.0.0.1:49832)
0928 21:49:18.908253 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:18.908296 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:18.908360 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:18.908769 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:18.908770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:18.908771 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:18.908882 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.908884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.910114 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.910115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.911153 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.911155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.912046 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.912047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.912113 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.912116 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:18.912166 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:49:18.912205 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"spinlock_wait_cycles":768,"thread_start_us":63,"threads_started":1}
I20260928 21:49:18.912964 7413 tablet_service.cc:1467] Processing CreateTablet for tablet 6f5d383fcf3b4e1384bbee132be48911 (DEFAULT_TABLE table=test-table [id=edc1989b86ac4c33810d5861196683ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:18.913002 7546 tablet_service.cc:1467] Processing CreateTablet for tablet 6f5d383fcf3b4e1384bbee132be48911 (DEFAULT_TABLE table=test-table [id=edc1989b86ac4c33810d5861196683ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:18.913223 7413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f5d383fcf3b4e1384bbee132be48911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:18.913326 7546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f5d383fcf3b4e1384bbee132be48911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:18.915514 7626 tablet_bootstrap.cc:492] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: Bootstrap starting.
I20260928 21:49:18.915560 7627 tablet_bootstrap.cc:492] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: Bootstrap starting.
I20260928 21:49:18.915887 7280 tablet_service.cc:1467] Processing CreateTablet for tablet 6f5d383fcf3b4e1384bbee132be48911 (DEFAULT_TABLE table=test-table [id=edc1989b86ac4c33810d5861196683ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:18.916169 7626 tablet_bootstrap.cc:654] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:18.916181 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f5d383fcf3b4e1384bbee132be48911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:18.916261 7627 tablet_bootstrap.cc:654] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:18.916757 7627 log.cc:824] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:18.916749 7626 log.cc:824] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:18.917404 7627 tablet_bootstrap.cc:492] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: No bootstrap required, opened a new log
I20260928 21:49:18.917464 7627 ts_tablet_manager.cc:1397] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:18.917608 7626 tablet_bootstrap.cc:492] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: No bootstrap required, opened a new log
I20260928 21:49:18.917681 7626 ts_tablet_manager.cc:1397] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:18.918797 7627 raft_consensus.cc:348] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 } } peers { permanent_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37511 } } peers { permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 } }
I20260928 21:49:18.918921 7627 raft_consensus.cc:374] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:18.918960 7627 raft_consensus.cc:729] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d2c7957d3a4872a4e25fde9c58f2a6, State: Initialized, Role: FOLLOWER
I20260928 21:49:18.919278 7627 consensus_queue.cc:260] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 } } peers { permanent_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37511 } } peers { permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 } }
I20260928 21:49:18.919590 7627 ts_tablet_manager.cc:1428] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:49:18.919499 7626 raft_consensus.cc:348] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 } } peers { permanent_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37511 } } peers { permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 } }
I20260928 21:49:18.919636 7612 heartbeater.cc:503] Master 127.3.0.254:43401 was elected leader, sending a full tablet report...
I20260928 21:49:18.919646 7626 raft_consensus.cc:374] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:18.919701 7626 raft_consensus.cc:729] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5d21ebe0c5436fbd4edf85664387f3, State: Initialized, Role: FOLLOWER
I20260928 21:49:18.919770 7630 tablet_bootstrap.cc:492] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb: Bootstrap starting.
I20260928 21:49:18.919809 7626 consensus_queue.cc:260] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 } } peers { permanent_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37511 } } peers { permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 } }
I20260928 21:49:18.920060 7479 heartbeater.cc:503] Master 127.3.0.254:43401 was elected leader, sending a full tablet report...
I20260928 21:49:18.920063 7626 ts_tablet_manager.cc:1428] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:49:18.920688 7630 tablet_bootstrap.cc:654] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:18.921181 7630 log.cc:824] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:18.921725 7630 tablet_bootstrap.cc:492] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb: No bootstrap required, opened a new log
I20260928 21:49:18.921775 7630 ts_tablet_manager.cc:1397] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:18.923110 7630 raft_consensus.cc:348] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 } } peers { permanent_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37511 } } peers { permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 } }
I20260928 21:49:18.923224 7630 raft_consensus.cc:374] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:18.923243 7630 raft_consensus.cc:729] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ec5984f54e41b0ac4aecbde24cddcb, State: Initialized, Role: FOLLOWER
I20260928 21:49:18.923339 7630 consensus_queue.cc:260] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 } } peers { permanent_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37511 } } peers { permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 } }
I20260928 21:49:18.923604 7630 ts_tablet_manager.cc:1428] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:49:18.923656 7346 heartbeater.cc:503] Master 127.3.0.254:43401 was elected leader, sending a full tablet report...
I20260928 21:49:18.968545 7631 raft_consensus.cc:482] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:18.968658 7631 raft_consensus.cc:504] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 } } peers { permanent_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37511 } } peers { permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 } }
I20260928 21:49:18.968935 7631 leader_election.cc:288] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c5d21ebe0c5436fbd4edf85664387f3 (127.3.0.194:38363), 05ec5984f54e41b0ac4aecbde24cddcb (127.3.0.193:35323)
I20260928 21:49:18.971483 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.969143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:38191 (local address 127.3.0.194:38363)
0928 21:49:18.969185 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.969186 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.969194 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:18.969246 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.969246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.969258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.969305 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:18.969441 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.969904 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.970812 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.971174 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.971233 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.971244 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.971388 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.971420 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:18.971491 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.969084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:38363 (local address 127.3.0.195:38191)
0928 21:49:18.969138 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:18.969149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:18.969190 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:18.969307 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:18.969308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:18.969309 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:18.969381 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.969383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.969981 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.969982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.970726 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.970728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.971248 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.971249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.971329 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.971332 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:18.971375 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:49:18.971413 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:49:18.971598 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.969093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:35323 (local address 127.3.0.195:53545)
0928 21:49:18.969253 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:18.969262 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:18.969320 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:18.969445 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:18.969446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:18.969447 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:18.969515 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.969517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.970109 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.970110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.970860 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:18.970862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.971361 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.971362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:18.971442 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.971445 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:18.971490 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:49:18.971524 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":116,"threads_started":1}
I20260928 21:49:18.971599 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.969148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:53545 (local address 127.3.0.193:35323)
0928 21:49:18.969189 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.969191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.969326 (+ 135us) server_negotiation.cc:407] Connection header received
0928 21:49:18.969376 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.969377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.969389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.969434 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:18.969573 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.970036 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.970944 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.971292 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.971348 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.971357 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.971504 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.971543 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:49:18.971807 7433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5d383fcf3b4e1384bbee132be48911" candidate_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" is_pre_election: true
I20260928 21:49:18.971912 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5d383fcf3b4e1384bbee132be48911" candidate_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" is_pre_election: true
I20260928 21:49:18.971957 7433 raft_consensus.cc:2454] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d2c7957d3a4872a4e25fde9c58f2a6 in term 0.
I20260928 21:49:18.972054 7300 raft_consensus.cc:2454] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94d2c7957d3a4872a4e25fde9c58f2a6 in term 0.
I20260928 21:49:18.972179 7502 leader_election.cc:302] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94d2c7957d3a4872a4e25fde9c58f2a6, 9c5d21ebe0c5436fbd4edf85664387f3; no voters:
I20260928 21:49:18.972349 7631 raft_consensus.cc:2781] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:18.972395 7631 raft_consensus.cc:482] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:18.972409 7631 raft_consensus.cc:3037] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:18.973089 7631 raft_consensus.cc:504] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 } } peers { permanent_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37511 } } peers { permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 } }
I20260928 21:49:18.973208 7631 leader_election.cc:288] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [CANDIDATE]: Term 1 election: Requested vote from peers 9c5d21ebe0c5436fbd4edf85664387f3 (127.3.0.194:38363), 05ec5984f54e41b0ac4aecbde24cddcb (127.3.0.193:35323)
I20260928 21:49:18.973337 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5d383fcf3b4e1384bbee132be48911" candidate_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05ec5984f54e41b0ac4aecbde24cddcb"
I20260928 21:49:18.973367 7433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f5d383fcf3b4e1384bbee132be48911" candidate_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5d21ebe0c5436fbd4edf85664387f3"
I20260928 21:49:18.973410 7300 raft_consensus.cc:3037] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:18.973429 7433 raft_consensus.cc:3037] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:18.974264 7433 raft_consensus.cc:2454] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d2c7957d3a4872a4e25fde9c58f2a6 in term 1.
I20260928 21:49:18.974264 7300 raft_consensus.cc:2454] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94d2c7957d3a4872a4e25fde9c58f2a6 in term 1.
I20260928 21:49:18.974428 7499 leader_election.cc:302] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05ec5984f54e41b0ac4aecbde24cddcb, 94d2c7957d3a4872a4e25fde9c58f2a6; no voters:
I20260928 21:49:18.974505 7631 raft_consensus.cc:2781] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:18.974640 7631 raft_consensus.cc:686] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 1 LEADER]: Becoming Leader. State: Replica: 94d2c7957d3a4872a4e25fde9c58f2a6, State: Running, Role: LEADER
I20260928 21:49:18.974721 7631 consensus_queue.cc:237] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 } } peers { permanent_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37511 } } peers { permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 } }
I20260928 21:49:18.975560 7155 catalog_manager.cc:5495] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94d2c7957d3a4872a4e25fde9c58f2a6 (127.3.0.195). New cstate: current_term: 1 leader_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94d2c7957d3a4872a4e25fde9c58f2a6" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 } health_report { overall_health: UNKNOWN } } }
W20260928 21:49:18.982215 7480 tablet.cc:2369] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:49:18.983173 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:18.981236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.3.0.195:37511)
0928 21:49:18.981272 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:18.981273 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:18.981280 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:49:18.981322 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:18.981323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:18.981340 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:18.981395 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:18.981494 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.981920 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.982616 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:18.982789 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:18.982838 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:18.982932 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:18.983019 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:18.983067 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:18.983082 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 21:49:18.983097 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:49:18.984668 7300 raft_consensus.cc:1261] T 6f5d383fcf3b4e1384bbee132be48911 P 05ec5984f54e41b0ac4aecbde24cddcb [term 1 FOLLOWER]: Refusing update from remote peer 94d2c7957d3a4872a4e25fde9c58f2a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:18.984680 7433 raft_consensus.cc:1261] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3 [term 1 FOLLOWER]: Refusing update from remote peer 94d2c7957d3a4872a4e25fde9c58f2a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:18.984994 7636 consensus_queue.cc:1035] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c5d21ebe0c5436fbd4edf85664387f3" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:18.985110 7631 consensus_queue.cc:1035] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05ec5984f54e41b0ac4aecbde24cddcb" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:18.987571 7641 mvcc.cc:205] Tried to move back new op lower bound from 7334429323199107072 to 7334429323161067520. Current Snapshot: MvccSnapshot[applied={T|T < 7334429323199107072}]
I20260928 21:49:18.987762 7640 mvcc.cc:205] Tried to move back new op lower bound from 7334429323199107072 to 7334429323161067520. Current Snapshot: MvccSnapshot[applied={T|T < 7334429323199107072 or (T in {7334429323199107072})}]
I20260928 21:49:18.989218 7154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:18.989306 7154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:18.989989 7154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adf4511c76324cbb8f926de6e678b9d9: Sending DeleteTablet for 3 replicas of tablet 6f5d383fcf3b4e1384bbee132be48911
I20260928 21:49:18.990293 7413 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f5d383fcf3b4e1384bbee132be48911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:18 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:54100
I20260928 21:49:18.990306 7546 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f5d383fcf3b4e1384bbee132be48911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:18 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:49832
I20260928 21:49:18.990315 7280 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f5d383fcf3b4e1384bbee132be48911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:18 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:44184
I20260928 21:49:18.990561 7649 tablet_replica.cc:330] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: stopping tablet replica
I20260928 21:49:18.990653 7649 raft_consensus.cc:2229] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:18.990784 7649 raft_consensus.cc:2258] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:18.990844 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 7218
I20260928 21:49:18.991843 7649 ts_tablet_manager.cc:1905] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:18.993186 7649 ts_tablet_manager.cc:1918] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:18.993247 7649 log.cc:1196] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/6f5d383fcf3b4e1384bbee132be48911
I20260928 21:49:18.993443 7649 ts_tablet_manager.cc:1939] T 6f5d383fcf3b4e1384bbee132be48911 P 94d2c7957d3a4872a4e25fde9c58f2a6: Deleting consensus metadata
I20260928 21:49:18.993775 7140 catalog_manager.cc:4841] TS 94d2c7957d3a4872a4e25fde9c58f2a6 (127.3.0.195:37511): tablet 6f5d383fcf3b4e1384bbee132be48911 (table test-table [id=edc1989b86ac4c33810d5861196683ed]) successfully deleted
I20260928 21:49:18.996558 7651 tablet_replica.cc:330] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: stopping tablet replica
I20260928 21:49:18.996658 7651 raft_consensus.cc:2229] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:18.996760 7651 raft_consensus.cc:2258] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:18.997107 7651 ts_tablet_manager.cc:1905] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:18.998211 7651 ts_tablet_manager.cc:1918] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:18.998272 7651 log.cc:1196] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/6f5d383fcf3b4e1384bbee132be48911
I20260928 21:49:18.998540 7651 ts_tablet_manager.cc:1939] T 6f5d383fcf3b4e1384bbee132be48911 P 9c5d21ebe0c5436fbd4edf85664387f3: Deleting consensus metadata
I20260928 21:49:18.998883 7143 catalog_manager.cc:4841] TS 9c5d21ebe0c5436fbd4edf85664387f3 (127.3.0.194:38363): tablet 6f5d383fcf3b4e1384bbee132be48911 (table test-table [id=edc1989b86ac4c33810d5861196683ed]) successfully deleted
W20260928 21:49:19.000141 7140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:35323 (error 108)
I20260928 21:49:19.000340 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 7351
I20260928 21:49:19.000911 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.000439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:35323 (local address 127.0.0.1:44186)
0928 21:49:19.000842 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.000879 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:35323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
W20260928 21:49:19.002517 7140 catalog_manager.cc:4584] TS 05ec5984f54e41b0ac4aecbde24cddcb (127.3.0.193:35323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f5d383fcf3b4e1384bbee132be48911: Network error: Client connection negotiation failed: client connection to 127.3.0.193:35323: connect: Connection refused (error 111)
I20260928 21:49:19.004299 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 7484
I20260928 21:49:19.007877 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 7125
2026-09-28T21:49:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1803 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T21:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:19Z Disabled control of system clock
I20260928 21:49:19.032200 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38331
--webserver_interface=127.3.0.254
--webserver_port=0
--builtin_ntp_servers=127.3.0.212:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:49:19.108718 7656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:19.108906 7656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:19.108925 7656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:19.109982 7656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:19.110015 7656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:19.110026 7656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:19.110037 7656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:19.110051 7656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:19.111292 7656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:38331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:19.111471 7656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:19.111683 7656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:19.115917 7662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:19.115932 7661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:19.115967 7656 server_base.cc:1034] running on GCE node
W20260928 21:49:19.115932 7664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:19.116247 7656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:19.116513 7656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:19.117651 7656 hybrid_clock.cc:648] HybridClock initialized: now 1790632159117634 us; error 30 us; skew 500 ppm
I20260928 21:49:19.118711 7656 webserver.cc:460] Webserver started at http://127.3.0.254:44239/ using document root <none> and password file <none>
I20260928 21:49:19.118907 7656 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:19.118953 7656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:19.119073 7656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:19.120028 7656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "2949c94f80964bf489e0db7d85fedb19"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "4011f13002bf3c60ebdf9fa69eeac624"
server_key_iv: "e7116dbd27b51854d908b9c2327a873a"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.120349 7656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "2949c94f80964bf489e0db7d85fedb19"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "4011f13002bf3c60ebdf9fa69eeac624"
server_key_iv: "e7116dbd27b51854d908b9c2327a873a"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.121511 7656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:19.122107 7670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.122251 7656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:19.122326 7656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "2949c94f80964bf489e0db7d85fedb19"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "4011f13002bf3c60ebdf9fa69eeac624"
server_key_iv: "e7116dbd27b51854d908b9c2327a873a"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.122390 7656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:19.125483 7656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:19.125720 7656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:19.125834 7656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:19.129328 7656 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:38331
I20260928 21:49:19.129357 7722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:38331 every 8 connection(s)
I20260928 21:49:19.129649 7656 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:19.130214 7723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:19.132830 7723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19: Bootstrap starting.
I20260928 21:49:19.133370 7723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:19.133646 7723 log.cc:824] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:19.134238 7723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19: No bootstrap required, opened a new log
I20260928 21:49:19.135844 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 7656
I20260928 21:49:19.135761 7723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2949c94f80964bf489e0db7d85fedb19" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38331 } }
I20260928 21:49:19.135944 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:19.135941 7723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:19.135974 7723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2949c94f80964bf489e0db7d85fedb19, State: Initialized, Role: FOLLOWER
I20260928 21:49:19.136116 3075 external_mini_cluster.cc:1428] Setting key 6a3bdb1a2895164ac1f5b58cb4c0ec0e
I20260928 21:49:19.136117 7723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2949c94f80964bf489e0db7d85fedb19" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38331 } }
I20260928 21:49:19.136193 7723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:19.136228 7723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:19.136272 7723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:19.136883 7723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2949c94f80964bf489e0db7d85fedb19" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38331 } }
I20260928 21:49:19.136977 7723 leader_election.cc:302] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2949c94f80964bf489e0db7d85fedb19; no voters:
I20260928 21:49:19.137113 7723 leader_election.cc:288] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:19.137151 7728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:19.137326 7728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [term 1 LEADER]: Becoming Leader. State: Replica: 2949c94f80964bf489e0db7d85fedb19, State: Running, Role: LEADER
I20260928 21:49:19.137457 7728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2949c94f80964bf489e0db7d85fedb19" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38331 } }
I20260928 21:49:19.137733 7723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:19.137765 7728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2949c94f80964bf489e0db7d85fedb19. Latest consensus state: current_term: 1 leader_uuid: "2949c94f80964bf489e0db7d85fedb19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2949c94f80964bf489e0db7d85fedb19" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38331 } } }
I20260928 21:49:19.137905 7728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:19.138044 7729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2949c94f80964bf489e0db7d85fedb19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2949c94f80964bf489e0db7d85fedb19" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38331 } } }
I20260928 21:49:19.138134 7732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:19.138152 7729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:19.138466 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.136563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.3.0.254:38331)
0928 21:49:19.136753 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.136763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.136797 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:49:19.136899 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.136905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.136938 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.137064 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.137723 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.137727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.137727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.137731 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.137774 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.137789 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.138059 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.138090 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.138146 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260928 21:49:19.138702 7732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:19.140535 7732 catalog_manager.cc:1347] Generated new cluster ID: 77843a8243bd4b2eacdcbb5f3c79d72e
I20260928 21:49:19.140581 7732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:49:19.153710 7732 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:19.153899 7732 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:19.161371 7732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19: Generated new TSK 0
I20260928 21:49:19.161557 7732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:49:19.168720 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38331
--builtin_ntp_servers=127.3.0.212:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:49:19.247628 7747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:19.247819 7747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:19.247839 7747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:19.249010 7747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:19.249048 7747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:19.249073 7747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:19.250376 7747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:19.250598 7747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:19.250864 7747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:19.256424 7752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:19.256554 7747 server_base.cc:1034] running on GCE node
W20260928 21:49:19.256419 7755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:19.256419 7753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:19.256883 7747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:19.257130 7747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:19.258275 7747 hybrid_clock.cc:648] HybridClock initialized: now 1790632159258235 us; error 53 us; skew 500 ppm
I20260928 21:49:19.259411 7747 webserver.cc:460] Webserver started at http://127.3.0.193:43495/ using document root <none> and password file <none>
I20260928 21:49:19.259617 7747 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:19.259661 7747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:19.259776 7747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:19.260869 7747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "a9718dcd6d1f4f5b91b7091c5803014b"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "0f02a2c93c5ac5f4293b157a42580716"
server_key_iv: "77becfca2b6118225957b0744fc26b2d"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.261191 7747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "a9718dcd6d1f4f5b91b7091c5803014b"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "0f02a2c93c5ac5f4293b157a42580716"
server_key_iv: "77becfca2b6118225957b0744fc26b2d"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.262409 7747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:19.263055 7761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.263195 7747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:19.263270 7747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "a9718dcd6d1f4f5b91b7091c5803014b"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "0f02a2c93c5ac5f4293b157a42580716"
server_key_iv: "77becfca2b6118225957b0744fc26b2d"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.263324 7747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:19.266435 7747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:19.266649 7747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:19.266736 7747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:19.266887 7747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:19.267119 7747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:19.267146 7747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.267164 7747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:19.267180 7747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.272233 7747 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:37359
I20260928 21:49:19.272289 7874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:37359 every 8 connection(s)
I20260928 21:49:19.272595 7747 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:49:19.272840 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 7747
I20260928 21:49:19.272945 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:19.273124 3075 external_mini_cluster.cc:1428] Setting key 252888e31670efde03113f5068722d3c
I20260928 21:49:19.274022 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38331
--builtin_ntp_servers=127.3.0.212:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 21:49:19.275792 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.273294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:60785 (local address 127.3.0.254:38331)
0928 21:49:19.273336 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.273339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.273618 (+ 279us) server_negotiation.cc:407] Connection header received
0928 21:49:19.274192 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.274193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.274201 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.274226 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.275566 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.275568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.275568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.275573 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.275589 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.275593 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.275645 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.275710 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.275725 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:49:19.275982 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.273291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38331 (local address 127.3.0.193:60785)
0928 21:49:19.273483 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.273523 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:19.274066 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:19.274416 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:19.274420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:19.274431 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:19.275425 (+ 994us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:49:19.275426 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:19.275456 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:49:19.275484 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:19.275647 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:19.275652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:19.275654 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:49:19.275720 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:49:19.275780 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260928 21:49:19.276386 7875 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38331
I20260928 21:49:19.276502 7875 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:19.276640 7875 heartbeater.cc:511] Master 127.3.0.254:38331 requested a full tablet report, sending...
I20260928 21:49:19.277078 7687 ts_manager.cc:194] Registered new tserver with Master: a9718dcd6d1f4f5b91b7091c5803014b (127.3.0.193:37359)
W20260928 21:49:19.352385 7878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:19.352607 7878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:19.352628 7878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:19.353766 7878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:19.353811 7878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:19.353838 7878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:19.355151 7878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:19.355361 7878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:19.355621 7878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:19.360152 7883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:19.360154 7884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:19.360146 7886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:19.360383 7878 server_base.cc:1034] running on GCE node
I20260928 21:49:19.360610 7878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:19.360829 7878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:19.361961 7878 hybrid_clock.cc:648] HybridClock initialized: now 1790632159361947 us; error 31 us; skew 500 ppm
I20260928 21:49:19.363162 7878 webserver.cc:460] Webserver started at http://127.3.0.194:33747/ using document root <none> and password file <none>
I20260928 21:49:19.363381 7878 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:19.363430 7878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:19.363564 7878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:19.364662 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "f8637a5fae164eb2b157492194a60447"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "dbfcf5dd8d14b8965d981bee7f1a9a39"
server_key_iv: "ec8cfbff889755bd566d587a57183486"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.364990 7878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "f8637a5fae164eb2b157492194a60447"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "dbfcf5dd8d14b8965d981bee7f1a9a39"
server_key_iv: "ec8cfbff889755bd566d587a57183486"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.366289 7878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:19.367051 7892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.367203 7878 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:19.367285 7878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "f8637a5fae164eb2b157492194a60447"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "dbfcf5dd8d14b8965d981bee7f1a9a39"
server_key_iv: "ec8cfbff889755bd566d587a57183486"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.367352 7878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:19.370582 7878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:19.370810 7878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:19.370924 7878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:19.371102 7878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:19.371361 7878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:19.371392 7878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.371417 7878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:19.371440 7878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.376633 7878 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:41579
I20260928 21:49:19.376684 8005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:41579 every 8 connection(s)
I20260928 21:49:19.376986 7878 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:49:19.378489 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 7878
I20260928 21:49:19.378576 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:19.378751 3075 external_mini_cluster.cc:1428] Setting key f1d6dff7a73e92bc77b231c45530b013
I20260928 21:49:19.379719 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.377586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:39321 (local address 127.3.0.254:38331)
0928 21:49:19.377659 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.377662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.377829 (+ 167us) server_negotiation.cc:407] Connection header received
0928 21:49:19.378248 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.378249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.378254 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.378321 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.379519 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.379520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.379520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.379523 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.379539 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.379542 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.379582 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.379657 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.379668 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:49:19.379735 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38331
--builtin_ntp_servers=127.3.0.212:43353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 21:49:19.379925 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.377561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38331 (local address 127.3.0.194:39321)
0928 21:49:19.377722 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.377753 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:19.378176 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:19.378397 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:19.378403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:19.378413 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:19.379381 (+ 968us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:49:19.379382 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:19.379410 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:49:19.379441 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:19.379585 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:19.379589 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:19.379592 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:19.379664 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:49:19.379728 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260928 21:49:19.381153 8006 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38331
I20260928 21:49:19.381229 8006 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:19.381361 8006 heartbeater.cc:511] Master 127.3.0.254:38331 requested a full tablet report, sending...
I20260928 21:49:19.381680 7687 ts_manager.cc:194] Registered new tserver with Master: f8637a5fae164eb2b157492194a60447 (127.3.0.194:41579)
W20260928 21:49:19.458094 8009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:19.458302 8009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:19.458329 8009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:19.459481 8009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:19.459518 8009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:19.459550 8009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:19.460924 8009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:43353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:19.461153 8009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:19.461417 8009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:19.465999 8014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:19.466076 8017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:19.466144 8009 server_base.cc:1034] running on GCE node
W20260928 21:49:19.465999 8015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:19.466399 8009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:19.466636 8009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:19.467773 8009 hybrid_clock.cc:648] HybridClock initialized: now 1790632159467759 us; error 28 us; skew 500 ppm
I20260928 21:49:19.468932 8009 webserver.cc:460] Webserver started at http://127.3.0.195:39949/ using document root <none> and password file <none>
I20260928 21:49:19.469151 8009 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:19.469198 8009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:19.469317 8009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:19.470316 8009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "5558cd4bbd8b43f780ce5b925e0bd095"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "32e6058b2da7c2d3a6bc860c6102ab1f"
server_key_iv: "cc246d1615f6e62f3a502aeb89fd1de1"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.470634 8009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "5558cd4bbd8b43f780ce5b925e0bd095"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "32e6058b2da7c2d3a6bc860c6102ab1f"
server_key_iv: "cc246d1615f6e62f3a502aeb89fd1de1"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.471822 8009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:19.472438 8023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.472605 8009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:19.472684 8009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "5558cd4bbd8b43f780ce5b925e0bd095"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "32e6058b2da7c2d3a6bc860c6102ab1f"
server_key_iv: "cc246d1615f6e62f3a502aeb89fd1de1"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.472749 8009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:19.475919 8009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:19.476161 8009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:19.476279 8009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:19.476498 8009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:19.476804 8009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:19.476835 8009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.476861 8009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:19.476886 8009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.482191 8009 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:46003
I20260928 21:49:19.482239 8136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:46003 every 8 connection(s)
I20260928 21:49:19.482555 8009 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:49:19.483923 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 8009
I20260928 21:49:19.484021 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:19.484217 3075 external_mini_cluster.cc:1428] Setting key 18cc2fa1078de8f98c96ac264b288135
I20260928 21:49:19.485562 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.483203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:49619 (local address 127.3.0.254:38331)
0928 21:49:19.483255 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.483257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.483481 (+ 224us) server_negotiation.cc:407] Connection header received
0928 21:49:19.484054 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.484056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.484062 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.484113 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.485372 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.485373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.485374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.485377 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.485397 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.485401 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.485442 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.485499 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.485511 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:49:19.485738 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.483199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38331 (local address 127.3.0.195:49619)
0928 21:49:19.483364 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.483397 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:19.483931 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:19.484189 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:19.484193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:19.484204 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:19.485238 (+ 1034us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:49:19.485239 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:19.485268 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:49:19.485297 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:19.485441 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:19.485443 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:19.485445 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:49:19.485499 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:49:19.485547 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260928 21:49:19.486126 8137 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38331
I20260928 21:49:19.486209 8137 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:19.486341 8137 heartbeater.cc:511] Master 127.3.0.254:38331 requested a full tablet report, sending...
I20260928 21:49:19.486646 7687 ts_manager.cc:194] Registered new tserver with Master: 5558cd4bbd8b43f780ce5b925e0bd095 (127.3.0.195:46003)
I20260928 21:49:19.495712 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:49:19.498018 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.496983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.3.0.254:38331)
0928 21:49:19.497033 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.497035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.497132 (+ 97us) server_negotiation.cc:407] Connection header received
0928 21:49:19.497198 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.497199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.497203 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.497245 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.497784 (+ 539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.497786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.497786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.497791 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.497807 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.497812 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.497858 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.497931 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.497946 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:49:19.499418 7687 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:19.500943 7687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:19.504506 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.502827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:37359 (local address 127.0.0.1:48906)
0928 21:49:19.503179 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.503189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:19.503246 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:19.503419 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:19.503424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:19.503431 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:19.504064 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:49:19.504065 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:19.504077 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:49:19.504101 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:19.504333 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:19.504340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:19.504343 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:19.504407 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:49:19.504419 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":47,"threads_started":1}
I20260928 21:49:19.504556 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.502932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.3.0.193:37359)
0928 21:49:19.503140 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.503144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.503305 (+ 161us) server_negotiation.cc:407] Connection header received
0928 21:49:19.503329 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.503331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.503341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.503399 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.504167 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.504175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.504175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.504180 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.504216 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.504248 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.504308 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.504424 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.504442 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
I20260928 21:49:19.504719 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.502930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.3.0.194:41579)
0928 21:49:19.503121 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.503126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.503303 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:49:19.503333 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.503335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.503343 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.503406 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.504355 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.504362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.504362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.504366 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.504407 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.504433 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.504509 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.504632 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.504647 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20260928 21:49:19.504743 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.503116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.3.0.195:46003)
0928 21:49:19.503252 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.503256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.503841 (+ 585us) server_negotiation.cc:407] Connection header received
0928 21:49:19.503884 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.503886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.503895 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.503963 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.504336 (+ 373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.504341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.504342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.504345 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.504373 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.504389 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.504438 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.504655 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.504670 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260928 21:49:19.504591 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.503024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:46003 (local address 127.0.0.1:37520)
0928 21:49:19.503766 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.503775 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:19.503829 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:19.503970 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:19.503971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:19.503973 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:19.504269 (+ 296us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:49:19.504270 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:19.504273 (+ 3us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:49:19.504281 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:19.504479 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:19.504480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:19.504481 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:49:19.504522 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 21:49:19.504533 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":65,"threads_started":1}
I20260928 21:49:19.504848 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.502822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:41579 (local address 127.0.0.1:38292)
0928 21:49:19.503145 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.503183 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:19.503244 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:19.503429 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:19.503430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:19.503433 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:19.504059 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:49:19.504060 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:19.504077 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:49:19.504278 (+ 201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:19.504532 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:19.504534 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:19.504534 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 21:49:19.504575 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 21:49:19.504588 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":79,"threads_started":1}
I20260928 21:49:19.505575 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 94e7a5bf54d342498dd634a9aaf62eb8 (DEFAULT_TABLE table=test-table [id=7e591c7b23784be88c09cf679115e718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:19.505586 7940 tablet_service.cc:1467] Processing CreateTablet for tablet 94e7a5bf54d342498dd634a9aaf62eb8 (DEFAULT_TABLE table=test-table [id=7e591c7b23784be88c09cf679115e718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:19.505576 7809 tablet_service.cc:1467] Processing CreateTablet for tablet 94e7a5bf54d342498dd634a9aaf62eb8 (DEFAULT_TABLE table=test-table [id=7e591c7b23784be88c09cf679115e718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:19.505885 7940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e7a5bf54d342498dd634a9aaf62eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:19.505893 7809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e7a5bf54d342498dd634a9aaf62eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:19.505904 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e7a5bf54d342498dd634a9aaf62eb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:19.508261 8151 tablet_bootstrap.cc:492] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: Bootstrap starting.
I20260928 21:49:19.508805 8151 tablet_bootstrap.cc:654] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:19.509012 8153 tablet_bootstrap.cc:492] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: Bootstrap starting.
I20260928 21:49:19.509109 8151 log.cc:824] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:19.509598 8153 tablet_bootstrap.cc:654] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:19.509825 8151 tablet_bootstrap.cc:492] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: No bootstrap required, opened a new log
I20260928 21:49:19.509893 8151 ts_tablet_manager.cc:1397] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:49:19.509888 8153 log.cc:824] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:19.510200 8152 tablet_bootstrap.cc:492] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b: Bootstrap starting.
I20260928 21:49:19.510516 8153 tablet_bootstrap.cc:492] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: No bootstrap required, opened a new log
I20260928 21:49:19.510579 8153 ts_tablet_manager.cc:1397] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:19.510859 8152 tablet_bootstrap.cc:654] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:19.511205 8152 log.cc:824] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:19.511966 8152 tablet_bootstrap.cc:492] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b: No bootstrap required, opened a new log
I20260928 21:49:19.512039 8152 ts_tablet_manager.cc:1397] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:19.512624 8153 raft_consensus.cc:348] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8637a5fae164eb2b157492194a60447" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41579 } } peers { permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 } } peers { permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 } }
I20260928 21:49:19.512588 8151 raft_consensus.cc:348] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8637a5fae164eb2b157492194a60447" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41579 } } peers { permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 } } peers { permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 } }
I20260928 21:49:19.512753 8151 raft_consensus.cc:374] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:19.512753 8153 raft_consensus.cc:374] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:19.512785 8151 raft_consensus.cc:729] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8637a5fae164eb2b157492194a60447, State: Initialized, Role: FOLLOWER
I20260928 21:49:19.512785 8153 raft_consensus.cc:729] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5558cd4bbd8b43f780ce5b925e0bd095, State: Initialized, Role: FOLLOWER
I20260928 21:49:19.512940 8153 consensus_queue.cc:260] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8637a5fae164eb2b157492194a60447" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41579 } } peers { permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 } } peers { permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 } }
I20260928 21:49:19.512950 8151 consensus_queue.cc:260] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8637a5fae164eb2b157492194a60447" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41579 } } peers { permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 } } peers { permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 } }
I20260928 21:49:19.513242 8006 heartbeater.cc:503] Master 127.3.0.254:38331 was elected leader, sending a full tablet report...
I20260928 21:49:19.513273 8153 ts_tablet_manager.cc:1428] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:49:19.513301 8137 heartbeater.cc:503] Master 127.3.0.254:38331 was elected leader, sending a full tablet report...
I20260928 21:49:19.513514 8151 ts_tablet_manager.cc:1428] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260928 21:49:19.514438 8152 raft_consensus.cc:348] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8637a5fae164eb2b157492194a60447" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41579 } } peers { permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 } } peers { permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 } }
I20260928 21:49:19.514569 8152 raft_consensus.cc:374] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:19.514590 8152 raft_consensus.cc:729] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9718dcd6d1f4f5b91b7091c5803014b, State: Initialized, Role: FOLLOWER
I20260928 21:49:19.514669 8152 consensus_queue.cc:260] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8637a5fae164eb2b157492194a60447" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41579 } } peers { permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 } } peers { permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 } }
I20260928 21:49:19.514865 8152 ts_tablet_manager.cc:1428] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:49:19.514914 7875 heartbeater.cc:503] Master 127.3.0.254:38331 was elected leader, sending a full tablet report...
W20260928 21:49:19.523140 7876 tablet.cc:2369] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:49:19.609400 8157 raft_consensus.cc:482] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:19.609527 8157 raft_consensus.cc:504] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8637a5fae164eb2b157492194a60447" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41579 } } peers { permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 } } peers { permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 } }
I20260928 21:49:19.609915 8157 leader_election.cc:288] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9718dcd6d1f4f5b91b7091c5803014b (127.3.0.193:37359), 5558cd4bbd8b43f780ce5b925e0bd095 (127.3.0.195:46003)
I20260928 21:49:19.611203 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.610207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:52667 (local address 127.3.0.193:37359)
0928 21:49:19.610280 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.610281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.610291 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:49:19.610318 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.610319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.610325 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.610369 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.611010 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.611011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.611011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.611015 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.611030 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.611034 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.611075 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.611124 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.611136 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:49:19.611203 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.610113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:37359 (local address 127.3.0.194:52667)
0928 21:49:19.610171 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.610184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:19.610244 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:19.610405 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:19.610407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:19.610409 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:19.610906 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:49:19.610906 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:19.610913 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:49:19.610924 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:19.611079 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:19.611079 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:19.611080 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:49:19.611123 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:49:19.611136 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 21:49:19.611339 8160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.610221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:46003 (local address 127.3.0.194:52467)
0928 21:49:19.610387 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.610395 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:19.610437 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:19.610612 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:19.610614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:19.610617 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:19.611026 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:49:19.611027 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:19.611034 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:49:19.611047 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:19.611208 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:19.611209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:19.611210 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:49:19.611249 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 21:49:19.611262 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":108,"threads_started":1}
I20260928 21:49:19.611330 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.610327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:52467 (local address 127.3.0.195:46003)
0928 21:49:19.610369 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.610371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.610449 (+ 78us) server_negotiation.cc:407] Connection header received
0928 21:49:19.610502 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.610503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.610508 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.610560 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.611131 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.611132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.611132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.611137 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.611153 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.611157 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.611206 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.611257 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.611269 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:49:19.611586 7829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e7a5bf54d342498dd634a9aaf62eb8" candidate_uuid: "f8637a5fae164eb2b157492194a60447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" is_pre_election: true
I20260928 21:49:19.611644 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e7a5bf54d342498dd634a9aaf62eb8" candidate_uuid: "f8637a5fae164eb2b157492194a60447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" is_pre_election: true
I20260928 21:49:19.611749 7829 raft_consensus.cc:2454] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8637a5fae164eb2b157492194a60447 in term 0.
I20260928 21:49:19.611771 8091 raft_consensus.cc:2454] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8637a5fae164eb2b157492194a60447 in term 0.
I20260928 21:49:19.611979 7896 leader_election.cc:302] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9718dcd6d1f4f5b91b7091c5803014b, f8637a5fae164eb2b157492194a60447; no voters:
I20260928 21:49:19.612083 8157 raft_consensus.cc:2781] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:19.612136 8157 raft_consensus.cc:482] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:19.612151 8157 raft_consensus.cc:3037] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:19.612838 8157 raft_consensus.cc:504] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8637a5fae164eb2b157492194a60447" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41579 } } peers { permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 } } peers { permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 } }
I20260928 21:49:19.612946 8157 leader_election.cc:288] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [CANDIDATE]: Term 1 election: Requested vote from peers a9718dcd6d1f4f5b91b7091c5803014b (127.3.0.193:37359), 5558cd4bbd8b43f780ce5b925e0bd095 (127.3.0.195:46003)
I20260928 21:49:19.613107 7829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e7a5bf54d342498dd634a9aaf62eb8" candidate_uuid: "f8637a5fae164eb2b157492194a60447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9718dcd6d1f4f5b91b7091c5803014b"
I20260928 21:49:19.613125 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e7a5bf54d342498dd634a9aaf62eb8" candidate_uuid: "f8637a5fae164eb2b157492194a60447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5558cd4bbd8b43f780ce5b925e0bd095"
I20260928 21:49:19.613190 8091 raft_consensus.cc:3037] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:19.613186 7829 raft_consensus.cc:3037] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:19.613848 8091 raft_consensus.cc:2454] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8637a5fae164eb2b157492194a60447 in term 1.
I20260928 21:49:19.613898 7829 raft_consensus.cc:2454] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8637a5fae164eb2b157492194a60447 in term 1.
I20260928 21:49:19.614061 7896 leader_election.cc:302] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9718dcd6d1f4f5b91b7091c5803014b, f8637a5fae164eb2b157492194a60447; no voters:
I20260928 21:49:19.614148 8157 raft_consensus.cc:2781] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:19.614292 8157 raft_consensus.cc:686] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 1 LEADER]: Becoming Leader. State: Replica: f8637a5fae164eb2b157492194a60447, State: Running, Role: LEADER
I20260928 21:49:19.614387 8157 consensus_queue.cc:237] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8637a5fae164eb2b157492194a60447" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41579 } } peers { permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 } } peers { permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 } }
I20260928 21:49:19.615190 7687 catalog_manager.cc:5495] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8637a5fae164eb2b157492194a60447 (127.3.0.194). New cstate: current_term: 1 leader_uuid: "f8637a5fae164eb2b157492194a60447" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8637a5fae164eb2b157492194a60447" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 } health_report { overall_health: UNKNOWN } } }
I20260928 21:49:19.618772 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.618092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.3.0.194:41579)
0928 21:49:19.618129 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.618130 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.618138 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:19.618183 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.618184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.618194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.618235 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.618636 (+ 401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.618637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.618637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.618640 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.618652 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.618655 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.618692 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.618719 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.618729 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:49:19.620298 8091 raft_consensus.cc:1261] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095 [term 1 FOLLOWER]: Refusing update from remote peer f8637a5fae164eb2b157492194a60447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:19.620319 7829 raft_consensus.cc:1261] T 94e7a5bf54d342498dd634a9aaf62eb8 P a9718dcd6d1f4f5b91b7091c5803014b [term 1 FOLLOWER]: Refusing update from remote peer f8637a5fae164eb2b157492194a60447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:19.620602 8157 consensus_queue.cc:1035] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5558cd4bbd8b43f780ce5b925e0bd095" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:19.620708 8161 consensus_queue.cc:1035] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9718dcd6d1f4f5b91b7091c5803014b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:19.623374 8164 mvcc.cc:205] Tried to move back new op lower bound from 7334429325802475520 to 7334429325781151744. Current Snapshot: MvccSnapshot[applied={T|T < 7334429325802475520 or (T in {7334429325802475520})}]
I20260928 21:49:19.623513 8165 mvcc.cc:205] Tried to move back new op lower bound from 7334429325802475520 to 7334429325781151744. Current Snapshot: MvccSnapshot[applied={T|T < 7334429325802475520}]
I20260928 21:49:19.625020 7687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:19.625108 7687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:19.625775 7687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2949c94f80964bf489e0db7d85fedb19: Sending DeleteTablet for 3 replicas of tablet 94e7a5bf54d342498dd634a9aaf62eb8
I20260928 21:49:19.626070 7809 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e7a5bf54d342498dd634a9aaf62eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:19 UTC) from {username='slave'} at 127.0.0.1:48906
I20260928 21:49:19.626065 7940 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e7a5bf54d342498dd634a9aaf62eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:19 UTC) from {username='slave'} at 127.0.0.1:38292
I20260928 21:49:19.626157 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e7a5bf54d342498dd634a9aaf62eb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:19 UTC) from {username='slave'} at 127.0.0.1:37520
I20260928 21:49:19.626348 8175 tablet_replica.cc:330] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: stopping tablet replica
I20260928 21:49:19.626430 8173 tablet_replica.cc:330] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: stopping tablet replica
I20260928 21:49:19.626515 8175 raft_consensus.cc:2229] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:19.626537 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 7747
I20260928 21:49:19.626521 8173 raft_consensus.cc:2229] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:19.627031 8173 raft_consensus.cc:2258] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:19.627183 8175 raft_consensus.cc:2258] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:19.627468 8173 ts_tablet_manager.cc:1905] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:19.627594 8175 ts_tablet_manager.cc:1905] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:19.629065 8173 ts_tablet_manager.cc:1918] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:19.629076 8175 ts_tablet_manager.cc:1918] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:19.629122 8173 log.cc:1196] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/94e7a5bf54d342498dd634a9aaf62eb8
I20260928 21:49:19.629133 8175 log.cc:1196] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/94e7a5bf54d342498dd634a9aaf62eb8
I20260928 21:49:19.629372 8175 ts_tablet_manager.cc:1939] T 94e7a5bf54d342498dd634a9aaf62eb8 P 5558cd4bbd8b43f780ce5b925e0bd095: Deleting consensus metadata
I20260928 21:49:19.629374 8173 ts_tablet_manager.cc:1939] T 94e7a5bf54d342498dd634a9aaf62eb8 P f8637a5fae164eb2b157492194a60447: Deleting consensus metadata
I20260928 21:49:19.629725 7673 catalog_manager.cc:4841] TS f8637a5fae164eb2b157492194a60447 (127.3.0.194:41579): tablet 94e7a5bf54d342498dd634a9aaf62eb8 (table test-table [id=7e591c7b23784be88c09cf679115e718]) successfully deleted
I20260928 21:49:19.629735 7674 catalog_manager.cc:4841] TS 5558cd4bbd8b43f780ce5b925e0bd095 (127.3.0.195:46003): tablet 94e7a5bf54d342498dd634a9aaf62eb8 (table test-table [id=7e591c7b23784be88c09cf679115e718]) successfully deleted
W20260928 21:49:19.632247 7674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:37359 (error 108)
I20260928 21:49:19.632766 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.632615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:37359 (local address 127.0.0.1:48908)
0928 21:49:19.632700 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.632734 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:37359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 21:49:19.632833 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 7878
W20260928 21:49:19.632884 7674 catalog_manager.cc:4584] TS a9718dcd6d1f4f5b91b7091c5803014b (127.3.0.193:37359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94e7a5bf54d342498dd634a9aaf62eb8: Network error: Client connection negotiation failed: client connection to 127.3.0.193:37359: connect: Connection refused (error 111)
I20260928 21:49:19.637569 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 8009
I20260928 21:49:19.641350 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 7656
2026-09-28T21:49:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (633 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 21:49:19.660342 3075 mini_oidc.cc:152] Starting JWKS server
I20260928 21:49:19.661568 3075 webserver.cc:460] Webserver started at https://127.0.0.1:35977/ using document root <none> and password file <none>
I20260928 21:49:19.661650 3075 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:49:19.661811 3075 webserver.cc:460] Webserver started at http://127.0.0.1:41363/ using document root <none> and password file <none>
2026-09-28T21:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:19Z Disabled control of system clock
I20260928 21:49:19.668741 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:34291
--webserver_interface=127.3.0.254
--webserver_port=0
--builtin_ntp_servers=127.3.0.212:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:34291
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35977/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/testchainca.cert with env {}
W20260928 21:49:19.747054 8182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:19.747255 8182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:19.747277 8182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:19.748487 8182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:19.748526 8182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:19.748538 8182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:19.748560 8182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:19.748584 8182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:49:19.748605 8182 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:49:19.748625 8182 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35977/jwks.json
I20260928 21:49:19.750155 8182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:34291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:34291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35977/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:19.750375 8182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:19.750572 8182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:19.755033 8187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:19.755057 8188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:19.755061 8190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:19.755209 8182 server_base.cc:1034] running on GCE node
I20260928 21:49:19.755438 8182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:19.755728 8182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:19.756875 8182 hybrid_clock.cc:648] HybridClock initialized: now 1790632159756859 us; error 34 us; skew 500 ppm
I20260928 21:49:19.758077 8182 webserver.cc:460] Webserver started at http://127.3.0.254:35355/ using document root <none> and password file <none>
I20260928 21:49:19.758335 8182 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:19.758453 8182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:19.758602 8182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:19.759636 8182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "f13daba616cf4c9ca3997100ff06a8f2"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "dfb1491a93f7716f5564048b6dfd5ba7"
server_key_iv: "9bdfb42394bffd7dbb5e7ad0b12c9802"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.759950 8182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "f13daba616cf4c9ca3997100ff06a8f2"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "dfb1491a93f7716f5564048b6dfd5ba7"
server_key_iv: "9bdfb42394bffd7dbb5e7ad0b12c9802"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.761160 8182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:49:19.761811 8196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.761948 8182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:19.762027 8182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "f13daba616cf4c9ca3997100ff06a8f2"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "dfb1491a93f7716f5564048b6dfd5ba7"
server_key_iv: "9bdfb42394bffd7dbb5e7ad0b12c9802"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.762082 8182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:19.773195 8182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:19.773492 8182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:19.773617 8182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:19.777180 8182 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:34291
I20260928 21:49:19.777215 8248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:34291 every 8 connection(s)
I20260928 21:49:19.777524 8182 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:19.778365 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:19.780591 8249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2: Bootstrap starting.
I20260928 21:49:19.781128 8249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:19.781356 8249 log.cc:824] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:19.781937 8249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2: No bootstrap required, opened a new log
I20260928 21:49:19.782670 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 8182
I20260928 21:49:19.782763 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:19.782948 3075 external_mini_cluster.cc:1428] Setting key f59b6330b9dd5b457f4e2ea147d7718d
I20260928 21:49:19.783890 8249 raft_consensus.cc:348] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13daba616cf4c9ca3997100ff06a8f2" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34291 } }
I20260928 21:49:19.784044 8249 raft_consensus.cc:374] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:19.784078 8249 raft_consensus.cc:729] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13daba616cf4c9ca3997100ff06a8f2, State: Initialized, Role: FOLLOWER
I20260928 21:49:19.784209 8249 consensus_queue.cc:260] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13daba616cf4c9ca3997100ff06a8f2" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34291 } }
I20260928 21:49:19.784287 8249 raft_consensus.cc:388] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:19.784312 8249 raft_consensus.cc:482] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:19.784355 8249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:19.784888 8249 raft_consensus.cc:504] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13daba616cf4c9ca3997100ff06a8f2" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34291 } }
I20260928 21:49:19.784976 8249 leader_election.cc:302] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f13daba616cf4c9ca3997100ff06a8f2; no voters:
I20260928 21:49:19.785120 8249 leader_election.cc:288] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:19.785162 8254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:19.785267 8254 raft_consensus.cc:686] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [term 1 LEADER]: Becoming Leader. State: Replica: f13daba616cf4c9ca3997100ff06a8f2, State: Running, Role: LEADER
I20260928 21:49:19.785367 8249 sys_catalog.cc:564] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:19.785344 8254 consensus_queue.cc:237] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13daba616cf4c9ca3997100ff06a8f2" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34291 } }
I20260928 21:49:19.785712 8254 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f13daba616cf4c9ca3997100ff06a8f2. Latest consensus state: current_term: 1 leader_uuid: "f13daba616cf4c9ca3997100ff06a8f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13daba616cf4c9ca3997100ff06a8f2" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34291 } } }
I20260928 21:49:19.785794 8254 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:19.785722 8256 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f13daba616cf4c9ca3997100ff06a8f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13daba616cf4c9ca3997100ff06a8f2" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34291 } } }
I20260928 21:49:19.785959 8256 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:19.786017 8260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:19.786496 8260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:19.788712 8260 catalog_manager.cc:1347] Generated new cluster ID: 219a3e43d63a4339b10432f0518ce350
I20260928 21:49:19.788748 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.783428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.3.0.254:34291)
0928 21:49:19.783606 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.783613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.783640 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:49:19.783849 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.783854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.783911 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.784019 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.784264 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:19.785784 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:19.786851 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:19.787330 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:19.787439 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:19.788046 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.788050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.788051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.788222 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.788254 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.788265 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.788320 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.788427 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.788536 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260928 21:49:19.788763 8260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:49:19.789660 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:49:19.796130 8260 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:19.796696 8260 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:19.798800 8260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f13daba616cf4c9ca3997100ff06a8f2: Generated new TSK 0
I20260928 21:49:19.798951 8260 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:49:19.823817 8279 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 21:49:19.823935 8279 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:49:19.824136 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.808924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.3.0.254:34291)
0928 21:49:19.808965 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.808966 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.809050 (+ 84us) server_negotiation.cc:407] Connection header received
0928 21:49:19.809151 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.809152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.809161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.809213 (+ 52us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:49:19.809342 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:19.809805 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:19.810909 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:19.811084 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:19.811132 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:19.811330 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:49:19.823918 (+ 12588us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35977/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:49:19.824072 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51662: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35977/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 21:49:19.824193 8253 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51662: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35977/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 21:49:19.824565 8278 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:49:19.808815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:34291 (local address 127.0.0.1:51662)
0928 21:49:19.808985 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.808996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:19.809065 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:19.809219 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:19.809220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:19.809222 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:49:19.809289 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:19.809291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:19.809890 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:19.809890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:19.810828 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:19.810832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:19.811144 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:19.811147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:19.811211 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:19.811233 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:49:19.824110 (+ 12877us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35977/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:49:19.824212 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.0.254:34291: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35977/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.0.254:34291: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35977/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 21:49:19.824949 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 8182
2026-09-28T21:49:19Z chronyd exiting
I20260928 21:49:19.859896 3075 test_util.cc:175] -----------------------------------------------
I20260928 21:49:19.859974 3075 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790632139349757-3075-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (200 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 21:49:19.860747 3075 mini_oidc.cc:152] Starting JWKS server
I20260928 21:49:19.861385 3075 webserver.cc:460] Webserver started at https://127.0.0.1:34799/ using document root <none> and password file <none>
I20260928 21:49:19.861420 3075 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:49:19.861523 3075 webserver.cc:460] Webserver started at http://127.0.0.1:39685/ using document root <none> and password file <none>
2026-09-28T21:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:19Z Disabled control of system clock
I20260928 21:49:19.868232 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:43873
--webserver_interface=127.3.0.254
--webserver_port=0
--builtin_ntp_servers=127.3.0.212:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:43873
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34799/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/testchainca.cert with env {}
W20260928 21:49:19.945266 8285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:19.945456 8285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:19.945477 8285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:19.946601 8285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:19.946637 8285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:19.946648 8285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:19.946660 8285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:19.946683 8285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:49:19.946702 8285 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:49:19.946722 8285 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34799/jwks.json
I20260928 21:49:19.948176 8285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:43873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:43873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34799/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:19.948390 8285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:19.948590 8285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:19.952911 8293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:19.952945 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:19.952976 8285 server_base.cc:1034] running on GCE node
W20260928 21:49:19.952983 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:19.953336 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:19.953591 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:19.954754 8285 hybrid_clock.cc:648] HybridClock initialized: now 1790632159954735 us; error 31 us; skew 500 ppm
I20260928 21:49:19.956005 8285 webserver.cc:460] Webserver started at http://127.3.0.254:40747/ using document root <none> and password file <none>
I20260928 21:49:19.956238 8285 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:19.956282 8285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:19.956408 8285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:19.957477 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "d548ae7161f0facd8466ec93c3ab9666"
server_key_iv: "1b4704d8dec04a989769b4106d5dcb5c"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.957784 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "d548ae7161f0facd8466ec93c3ab9666"
server_key_iv: "1b4704d8dec04a989769b4106d5dcb5c"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.958977 8285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:19.959637 8299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:19.959784 8285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:19.959869 8285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9"
format_stamp: "Formatted at 2026-09-28 21:49:19 on dist-test-slave-jrck"
server_key: "d548ae7161f0facd8466ec93c3ab9666"
server_key_iv: "1b4704d8dec04a989769b4106d5dcb5c"
server_key_version: "encryptionkey@0"
I20260928 21:49:19.959928 8285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:19.968032 8285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:19.968307 8285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:19.968423 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:19.971844 8285 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:43873
I20260928 21:49:19.971875 8351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:43873 every 8 connection(s)
I20260928 21:49:19.972162 8285 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:19.973057 8352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:19.975195 8352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9: Bootstrap starting.
I20260928 21:49:19.975734 8352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:19.975981 8352 log.cc:824] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:19.976696 8352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9: No bootstrap required, opened a new log
I20260928 21:49:19.978221 8352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43873 } }
I20260928 21:49:19.978358 8352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:19.978379 8352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ba44e4dc994fcdac14d8d8aa00b0f9, State: Initialized, Role: FOLLOWER
I20260928 21:49:19.978523 8352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43873 } }
I20260928 21:49:19.978600 8352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:19.978622 8352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:19.978654 8352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:19.979157 8352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43873 } }
I20260928 21:49:19.979242 8352 leader_election.cc:302] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34ba44e4dc994fcdac14d8d8aa00b0f9; no voters:
I20260928 21:49:19.979430 8352 leader_election.cc:288] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:19.979518 8355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:19.979708 8352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:19.979738 8355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [term 1 LEADER]: Becoming Leader. State: Replica: 34ba44e4dc994fcdac14d8d8aa00b0f9, State: Running, Role: LEADER
I20260928 21:49:19.979846 8355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43873 } }
I20260928 21:49:19.980176 8355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ba44e4dc994fcdac14d8d8aa00b0f9. Latest consensus state: current_term: 1 leader_uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43873 } } }
I20260928 21:49:19.980242 8355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:19.980564 8356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ba44e4dc994fcdac14d8d8aa00b0f9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43873 } } }
I20260928 21:49:19.980621 8356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:19.980974 8364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:19.981410 8364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:19.981813 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 8285
I20260928 21:49:19.981896 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:19.982088 3075 external_mini_cluster.cc:1428] Setting key ff62845b4bdad0e7ae4cc6b9e981bc4c
I20260928 21:49:19.982982 8364 catalog_manager.cc:1347] Generated new cluster ID: a0d53f88b4dd4a53a96da7a67886fbb7
I20260928 21:49:19.983031 8364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:49:19.987272 8364 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:19.987720 8364 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:19.988435 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.982467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.3.0.254:43873)
0928 21:49:19.982644 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.982653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.982684 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:49:19.982798 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.982805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.982875 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.983014 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:19.983350 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:19.984738 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:19.985867 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:19.986512 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:19.986636 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:19.987544 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:19.987548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:19.987549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:19.987819 (+ 270us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:19.987865 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:19.987881 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:19.987938 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:19.988126 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 21:49:19.988217 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260928 21:49:19.989710 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:49:19.991045 8364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34ba44e4dc994fcdac14d8d8aa00b0f9: Generated new TSK 0
I20260928 21:49:19.991199 8364 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:49:20.020416 8382 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 21:49:20.020530 8382 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:49:20.020845 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:19.999023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.3.0.254:43873)
0928 21:49:19.999073 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:19.999075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:19.999169 (+ 94us) server_negotiation.cc:407] Connection header received
0928 21:49:19.999230 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:19.999231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:19.999241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:19.999302 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:49:19.999417 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:19.999918 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.000730 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.000932 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.000985 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:20.001138 (+ 153us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:49:20.020605 (+ 19467us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:49:20.020772 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 21:49:20.020915 8374 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51136: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 21:49:20.020974 8381 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:49:19.998960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:43873 (local address 127.0.0.1:51136)
0928 21:49:19.999113 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:19.999122 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:19.999174 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:19.999305 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:19.999306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:19.999308 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:49:19.999360 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:19.999361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.000006 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.000007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:20.000657 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:20.000659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.000989 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.000993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:20.001048 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:20.001053 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:49:20.020744 (+ 19691us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:49:20.020858 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.0.254:43873: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260928 21:49:20.021224 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 8285
2026-09-28T21:49:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (197 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 21:49:20.058620 3075 mini_oidc.cc:152] Starting JWKS server
I20260928 21:49:20.059552 3075 webserver.cc:460] Webserver started at https://127.0.0.1:45069/ using document root <none> and password file <none>
I20260928 21:49:20.059613 3075 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:49:20.059767 3075 webserver.cc:460] Webserver started at http://127.0.0.1:46139/ using document root <none> and password file <none>
2026-09-28T21:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:20Z Disabled control of system clock
I20260928 21:49:20.067502 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:32903
--webserver_interface=127.3.0.254
--webserver_port=0
--builtin_ntp_servers=127.3.0.212:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:32903
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45069/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:49:20.145987 8388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:20.146179 8388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:20.146206 8388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:20.147816 8388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:20.147871 8388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:20.147889 8388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:20.147914 8388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:20.147938 8388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:49:20.147969 8388 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:49:20.147992 8388 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45069/jwks.json
I20260928 21:49:20.149446 8388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:32903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:32903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45069/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:20.149657 8388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:20.149926 8388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:20.154322 8396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:20.154316 8394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:20.154563 8388 server_base.cc:1034] running on GCE node
W20260928 21:49:20.154316 8393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:20.154796 8388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:20.155063 8388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:20.156195 8388 hybrid_clock.cc:648] HybridClock initialized: now 1790632160156183 us; error 24 us; skew 500 ppm
I20260928 21:49:20.157284 8388 webserver.cc:460] Webserver started at http://127.3.0.254:35089/ using document root <none> and password file <none>
I20260928 21:49:20.157502 8388 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:20.157550 8388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:20.157668 8388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:20.158659 8388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "98398479733240059b24754cdd8928ad"
format_stamp: "Formatted at 2026-09-28 21:49:20 on dist-test-slave-jrck"
server_key: "7484117ff9e4f99e7125fac951baa29a"
server_key_iv: "36d4d5a917b09c36b108fd3c818d0417"
server_key_version: "encryptionkey@0"
I20260928 21:49:20.159003 8388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "98398479733240059b24754cdd8928ad"
format_stamp: "Formatted at 2026-09-28 21:49:20 on dist-test-slave-jrck"
server_key: "7484117ff9e4f99e7125fac951baa29a"
server_key_iv: "36d4d5a917b09c36b108fd3c818d0417"
server_key_version: "encryptionkey@0"
I20260928 21:49:20.160147 8388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:49:20.160791 8402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:20.160918 8388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:20.161011 8388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "98398479733240059b24754cdd8928ad"
format_stamp: "Formatted at 2026-09-28 21:49:20 on dist-test-slave-jrck"
server_key: "7484117ff9e4f99e7125fac951baa29a"
server_key_iv: "36d4d5a917b09c36b108fd3c818d0417"
server_key_version: "encryptionkey@0"
I20260928 21:49:20.161083 8388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:20.171080 8388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:20.171350 8388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:20.171449 8388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:20.174983 8388 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:32903
I20260928 21:49:20.175004 8454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:32903 every 8 connection(s)
I20260928 21:49:20.175298 8388 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:20.175879 8455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:20.178378 8455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad: Bootstrap starting.
I20260928 21:49:20.178875 8455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:20.179111 8455 log.cc:824] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:20.179673 8455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad: No bootstrap required, opened a new log
I20260928 21:49:20.181239 8455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98398479733240059b24754cdd8928ad" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 32903 } }
I20260928 21:49:20.181377 8455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:20.181375 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 8388
I20260928 21:49:20.181409 8455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98398479733240059b24754cdd8928ad, State: Initialized, Role: FOLLOWER
I20260928 21:49:20.181449 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:20.181576 8455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98398479733240059b24754cdd8928ad" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 32903 } }
I20260928 21:49:20.181638 3075 external_mini_cluster.cc:1428] Setting key 5eae3b55d3ced3b45b0fd0e37b9088b0
I20260928 21:49:20.181641 8455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:20.181672 8455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:20.181718 8455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:20.182363 8455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98398479733240059b24754cdd8928ad" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 32903 } }
I20260928 21:49:20.182467 8455 leader_election.cc:302] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98398479733240059b24754cdd8928ad; no voters:
I20260928 21:49:20.182639 8455 leader_election.cc:288] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:20.182677 8460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:20.182801 8460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [term 1 LEADER]: Becoming Leader. State: Replica: 98398479733240059b24754cdd8928ad, State: Running, Role: LEADER
I20260928 21:49:20.182878 8455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:20.182889 8460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98398479733240059b24754cdd8928ad" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 32903 } }
I20260928 21:49:20.183238 8460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98398479733240059b24754cdd8928ad. Latest consensus state: current_term: 1 leader_uuid: "98398479733240059b24754cdd8928ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98398479733240059b24754cdd8928ad" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 32903 } } }
I20260928 21:49:20.183308 8460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:20.183476 8461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98398479733240059b24754cdd8928ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98398479733240059b24754cdd8928ad" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 32903 } } }
I20260928 21:49:20.183528 8461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:20.183622 8467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:20.184063 8467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:20.185516 8467 catalog_manager.cc:1347] Generated new cluster ID: 01b84b86d2224a9fbaba492be083c2e1
I20260928 21:49:20.185597 8467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:49:20.187261 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:20.182043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.3.0.254:32903)
0928 21:49:20.182252 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:20.182261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:20.182295 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:49:20.182423 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:20.182431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:20.182516 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:20.182662 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:20.183058 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.184746 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.185519 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.185989 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.186095 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:20.186687 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:20.186690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:20.186691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:20.186851 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:20.186882 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:20.186892 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:20.186946 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:20.187027 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 21:49:20.187106 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260928 21:49:20.189268 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:49:20.199887 8467 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:20.200613 8467 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:20.206794 8467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98398479733240059b24754cdd8928ad: Generated new TSK 0
I20260928 21:49:20.206971 8467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:49:20.217365 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:20.213993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.3.0.254:32903)
0928 21:49:20.214036 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:20.214038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:20.214093 (+ 55us) server_negotiation.cc:407] Connection header received
0928 21:49:20.214179 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:20.214179 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:20.214189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:20.214245 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:49:20.214358 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.214818 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.215896 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.216090 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.216132 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:20.216287 (+ 155us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:49:20.217177 (+ 890us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:49:20.217302 (+ 125us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 21:49:20.217422 8459 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58600: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 21:49:20.217720 8484 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:49:20.213891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:32903 (local address 127.0.0.1:58600)
0928 21:49:20.214034 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:20.214045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:20.214112 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:20.214246 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:20.214247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:20.214249 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:49:20.214306 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:20.214308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.214877 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.214879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:20.215826 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:20.215829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.216144 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.216147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:20.216205 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:20.216210 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:49:20.217279 (+ 1069us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:49:20.217377 (+ 98us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.0.254:32903: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45069/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":97,"threads_started":1}
I20260928 21:49:20.218005 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 8388
E20260928 21:49:20.227771 8485 webserver.cc:533] Webserver: sslize failed:
E20260928 21:49:20.227843 8485 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T21:49:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (196 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T21:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:20Z Disabled control of system clock
I20260928 21:49:20.261654 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:42387
--webserver_interface=127.3.0.254
--webserver_port=0
--builtin_ntp_servers=127.3.0.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:42387
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/testchainca.cert with env {}
W20260928 21:49:20.338651 8489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:20.338861 8489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:20.338891 8489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:20.340055 8489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:20.340101 8489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:20.340116 8489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:20.340132 8489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:20.340147 8489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:49:20.340162 8489 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:49:20.340180 8489 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 21:49:20.341704 8489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:42387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:20.341964 8489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:20.342164 8489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:20.346707 8497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:20.346900 8489 server_base.cc:1034] running on GCE node
W20260928 21:49:20.346714 8495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:20.346707 8494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:20.347218 8489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:20.347494 8489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:20.348621 8489 hybrid_clock.cc:648] HybridClock initialized: now 1790632160348608 us; error 30 us; skew 500 ppm
I20260928 21:49:20.349779 8489 webserver.cc:460] Webserver started at http://127.3.0.254:41845/ using document root <none> and password file <none>
I20260928 21:49:20.349992 8489 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:20.350037 8489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:20.350159 8489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:20.351177 8489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "8bf69f80962949c8ab131a2e179c3392"
format_stamp: "Formatted at 2026-09-28 21:49:20 on dist-test-slave-jrck"
server_key: "16039597d15ec33a0a70d1f2b4ca1b18"
server_key_iv: "370ec775e294f65025e665b481bff82c"
server_key_version: "encryptionkey@0"
I20260928 21:49:20.351505 8489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "8bf69f80962949c8ab131a2e179c3392"
format_stamp: "Formatted at 2026-09-28 21:49:20 on dist-test-slave-jrck"
server_key: "16039597d15ec33a0a70d1f2b4ca1b18"
server_key_iv: "370ec775e294f65025e665b481bff82c"
server_key_version: "encryptionkey@0"
I20260928 21:49:20.352667 8489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:49:20.353384 8503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:20.353529 8489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:20.353606 8489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "8bf69f80962949c8ab131a2e179c3392"
format_stamp: "Formatted at 2026-09-28 21:49:20 on dist-test-slave-jrck"
server_key: "16039597d15ec33a0a70d1f2b4ca1b18"
server_key_iv: "370ec775e294f65025e665b481bff82c"
server_key_version: "encryptionkey@0"
I20260928 21:49:20.353672 8489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:20.367628 8489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:20.367880 8489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:20.367996 8489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:20.371590 8489 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:42387
I20260928 21:49:20.371618 8555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:42387 every 8 connection(s)
I20260928 21:49:20.371912 8489 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:20.372799 8556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:20.374935 8556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392: Bootstrap starting.
I20260928 21:49:20.375304 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 8489
I20260928 21:49:20.375392 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:20.375491 8556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:20.375541 3075 external_mini_cluster.cc:1428] Setting key 3c29bfbdfb74e910205afbd89ee03132
I20260928 21:49:20.375753 8556 log.cc:824] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:20.376497 8556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392: No bootstrap required, opened a new log
I20260928 21:49:20.378919 8556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf69f80962949c8ab131a2e179c3392" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42387 } }
I20260928 21:49:20.379053 8556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:20.379074 8556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf69f80962949c8ab131a2e179c3392, State: Initialized, Role: FOLLOWER
I20260928 21:49:20.379185 8556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf69f80962949c8ab131a2e179c3392" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42387 } }
I20260928 21:49:20.379253 8556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:20.379284 8556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:20.379333 8556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:20.380048 8556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf69f80962949c8ab131a2e179c3392" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42387 } }
I20260928 21:49:20.380164 8556 leader_election.cc:302] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bf69f80962949c8ab131a2e179c3392; no voters:
I20260928 21:49:20.380366 8556 leader_election.cc:288] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:20.380407 8561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:20.380564 8561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [term 1 LEADER]: Becoming Leader. State: Replica: 8bf69f80962949c8ab131a2e179c3392, State: Running, Role: LEADER
I20260928 21:49:20.380640 8556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:20.380641 8561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf69f80962949c8ab131a2e179c3392" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42387 } }
I20260928 21:49:20.381301 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bf69f80962949c8ab131a2e179c3392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf69f80962949c8ab131a2e179c3392" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42387 } } }
I20260928 21:49:20.381378 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:20.381503 8563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bf69f80962949c8ab131a2e179c3392. Latest consensus state: current_term: 1 leader_uuid: "8bf69f80962949c8ab131a2e179c3392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bf69f80962949c8ab131a2e179c3392" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42387 } } }
I20260928 21:49:20.381606 8563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:20.381872 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:20.375996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.3.0.254:42387)
0928 21:49:20.376196 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:20.376205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:20.376236 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:49:20.376825 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:20.376835 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:20.376911 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:20.377061 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:20.377230 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.378613 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.379504 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.380102 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.380221 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:20.380915 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:20.380919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:20.380919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:49:20.381132 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:20.381181 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:49:20.381199 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:20.381530 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:20.381564 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 21:49:20.381653 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 21:49:20.382083 8570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:20.382669 8570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:20.382840 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:49:20.384280 8570 catalog_manager.cc:1347] Generated new cluster ID: cc6269780e814685b130ccb56cadfd50
I20260928 21:49:20.384330 8570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:49:20.391429 8570 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:20.392045 8570 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:20.396260 8570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bf69f80962949c8ab131a2e179c3392: Generated new TSK 0
I20260928 21:49:20.396442 8570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:49:20.425248 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:20.404852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.3.0.254:42387)
0928 21:49:20.404896 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:20.404898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:20.404939 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:49:20.405022 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:20.405023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:20.405034 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:20.405091 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:49:20.405210 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.405712 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.406444 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.406644 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.406697 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:20.406860 (+ 163us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:49:20.425011 (+ 18151us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:49:20.425173 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43320: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 21:49:20.425329 8560 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43320: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 21:49:20.425354 8585 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:49:20.404748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:42387 (local address 127.0.0.1:43320)
0928 21:49:20.404880 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:20.404891 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:20.404958 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:20.405094 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:20.405095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:20.405097 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:49:20.405149 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:20.405151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.405790 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.405791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:20.406370 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:20.406372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:20.406700 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:20.406704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:20.406768 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:20.406774 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:49:20.425152 (+ 18378us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:49:20.425265 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.0.254:42387: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260928 21:49:20.425644 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 8489
2026-09-28T21:49:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (187 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:20 dist-test-slave-jrck krb5kdc[8597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:20 dist-test-slave-jrck krb5kdc[8597](info): set up 2 sockets
Sep 28 21:49:20 dist-test-slave-jrck krb5kdc[8597](info): commencing operation
krb5kdc: starting...
W20260928 21:49:21.556289 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:21 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:21.661245 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33283
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:33283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:21.740391 8610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:21.740661 8610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:21.740694 8610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:21.741900 8610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:21.741943 8610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:21.741962 8610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:21.741979 8610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:21.741993 8610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:21.743490 8610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:33283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:21.743883 8610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:21.744109 8610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:21.748807 8618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:21.748800 8615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:21.748883 8610 server_base.cc:1034] running on GCE node
W20260928 21:49:21.748801 8616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:21.749257 8610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:21.749548 8610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:21.750684 8610 hybrid_clock.cc:648] HybridClock initialized: now 1790632161750667 us; error 35 us; skew 500 ppm
Sep 28 21:49:21 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632161, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:21.757027 8610 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:49:21.757437 8610 webserver.cc:460] Webserver started at http://127.3.0.254:39995/ using document root <none> and password file <none>
I20260928 21:49:21.757649 8610 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:21.757697 8610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:21.757822 8610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:21.758858 8610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "2e21961bf383452cae1c761fba8817e0"
format_stamp: "Formatted at 2026-09-28 21:49:21 on dist-test-slave-jrck"
server_key: "a4395e0645dde247ee1ff423c3f7019e"
server_key_iv: "a71c4489cd88f31c7e91f4278c657a0c"
server_key_version: "encryptionkey@0"
I20260928 21:49:21.759204 8610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "2e21961bf383452cae1c761fba8817e0"
format_stamp: "Formatted at 2026-09-28 21:49:21 on dist-test-slave-jrck"
server_key: "a4395e0645dde247ee1ff423c3f7019e"
server_key_iv: "a71c4489cd88f31c7e91f4278c657a0c"
server_key_version: "encryptionkey@0"
I20260928 21:49:21.760389 8610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:21.761066 8625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:21.761214 8610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:21.761294 8610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "2e21961bf383452cae1c761fba8817e0"
format_stamp: "Formatted at 2026-09-28 21:49:21 on dist-test-slave-jrck"
server_key: "a4395e0645dde247ee1ff423c3f7019e"
server_key_iv: "a71c4489cd88f31c7e91f4278c657a0c"
server_key_version: "encryptionkey@0"
I20260928 21:49:21.761360 8610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:21.770087 8610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:21.771759 8610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:21.771886 8610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:21.775592 8610 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:33283
I20260928 21:49:21.775627 8677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:33283 every 8 connection(s)
I20260928 21:49:21.775944 8610 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:21.776770 8678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:21.778916 8678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0: Bootstrap starting.
I20260928 21:49:21.779434 8678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:21.779685 8678 log.cc:824] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:21.780270 8678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0: No bootstrap required, opened a new log
I20260928 21:49:21.781822 8678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e21961bf383452cae1c761fba8817e0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33283 } }
I20260928 21:49:21.781958 8678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:21.782008 8678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e21961bf383452cae1c761fba8817e0, State: Initialized, Role: FOLLOWER
I20260928 21:49:21.782147 8678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e21961bf383452cae1c761fba8817e0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33283 } }
I20260928 21:49:21.782205 8678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:21.782226 8678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:21.782269 8678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:21.782748 8678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e21961bf383452cae1c761fba8817e0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33283 } }
I20260928 21:49:21.782842 8678 leader_election.cc:302] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e21961bf383452cae1c761fba8817e0; no voters:
I20260928 21:49:21.782998 8678 leader_election.cc:288] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:21.783041 8681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:21.783227 8678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:21.783372 8681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2e21961bf383452cae1c761fba8817e0, State: Running, Role: LEADER
I20260928 21:49:21.783490 8681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e21961bf383452cae1c761fba8817e0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33283 } }
I20260928 21:49:21.783854 8682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e21961bf383452cae1c761fba8817e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e21961bf383452cae1c761fba8817e0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33283 } } }
I20260928 21:49:21.783869 8683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e21961bf383452cae1c761fba8817e0. Latest consensus state: current_term: 1 leader_uuid: "2e21961bf383452cae1c761fba8817e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e21961bf383452cae1c761fba8817e0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33283 } } }
I20260928 21:49:21.783957 8682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:21.783962 8683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:21.784137 8690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:21.784646 8690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:21.785641 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 8610
I20260928 21:49:21.785727 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:21.785888 3075 external_mini_cluster.cc:1428] Setting key 8e13742c6ff7c86dc435de09e9dd2bb4
I20260928 21:49:21.786408 8690 catalog_manager.cc:1347] Generated new cluster ID: 956858877da84661a101891961d95086
I20260928 21:49:21.786459 8690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:49:21 dist-test-slave-jrck krb5kdc[8597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:21.796084 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:21.786300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.3.0.254:33283)
0928 21:49:21.786593 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:21.786601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:21.786623 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:49:21.786716 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:21.786722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:21.786762 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:21.786874 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:21.787477 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:21.788392 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:21.789130 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:21.789768 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:21.790158 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:21.792517 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:21.792522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:21.792522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:21.792533 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:21.794282 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:21.794667 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:21.794670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:21.794670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:21.794762 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:21.794928 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:21.794928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:21.794929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:21.795209 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:21.795666 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:21.795732 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 21:49:21.795856 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":70,"threads_started":1}
I20260928 21:49:21.830035 8690 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:21.830566 8690 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:21.838035 8690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e21961bf383452cae1c761fba8817e0: Generated new TSK 0
I20260928 21:49:21.838217 8690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:49:21.862954 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33283
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:21.941761 8704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:21.941959 8704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:21.941980 8704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:21.943128 8704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:21.943171 8704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:21.943198 8704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:21.944674 8704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:21.944900 8704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:21.945129 8704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:21.950042 8710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:21.950102 8704 server_base.cc:1034] running on GCE node
W20260928 21:49:21.950069 8709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:21.950042 8712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:21.950466 8704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:21.950675 8704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:21.951818 8704 hybrid_clock.cc:648] HybridClock initialized: now 1790632161951807 us; error 33 us; skew 500 ppm
Sep 28 21:49:21 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632161, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:21.954470 8704 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:49:21.954878 8704 webserver.cc:460] Webserver started at http://127.3.0.193:37901/ using document root <none> and password file <none>
I20260928 21:49:21.955046 8704 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:21.955084 8704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:21.955171 8704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:21.956122 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "ca35a12106c74fbea52d84556468fc05"
format_stamp: "Formatted at 2026-09-28 21:49:21 on dist-test-slave-jrck"
server_key: "ff88858d10ccdc93defd26e200218f85"
server_key_iv: "5051e054a9d53aac3093a826a4638916"
server_key_version: "encryptionkey@0"
I20260928 21:49:21.956418 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "ca35a12106c74fbea52d84556468fc05"
format_stamp: "Formatted at 2026-09-28 21:49:21 on dist-test-slave-jrck"
server_key: "ff88858d10ccdc93defd26e200218f85"
server_key_iv: "5051e054a9d53aac3093a826a4638916"
server_key_version: "encryptionkey@0"
I20260928 21:49:21.957724 8704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:21.958565 8719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:21.958709 8704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:21.958822 8704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "ca35a12106c74fbea52d84556468fc05"
format_stamp: "Formatted at 2026-09-28 21:49:21 on dist-test-slave-jrck"
server_key: "ff88858d10ccdc93defd26e200218f85"
server_key_iv: "5051e054a9d53aac3093a826a4638916"
server_key_version: "encryptionkey@0"
I20260928 21:49:21.958881 8704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:21.969398 8704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:21.971107 8704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:21.971212 8704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:21.971402 8704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:21.971679 8704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:21.971710 8704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:21.971729 8704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:21.971740 8704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:21.977139 8704 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:42225
I20260928 21:49:21.977180 8832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:42225 every 8 connection(s)
I20260928 21:49:21.977492 8704 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:49:21 dist-test-slave-jrck krb5kdc[8597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632161, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:21.986294 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:21.978166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:35205 (local address 127.3.0.254:33283)
0928 21:49:21.978245 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:21.978247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:21.978424 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:49:21.979349 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:21.979350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:21.979361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:21.979431 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:21.980016 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:21.980579 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:21.981603 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:21.981807 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:21.981878 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:21.983977 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:21.983978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:21.983979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:21.983986 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:21.985254 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:21.985614 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:21.985615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:21.985615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:21.985661 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:21.985829 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:21.985830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:21.985830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:21.985921 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:21.985966 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:21.986165 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 21:49:21.986228 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:49:21.986464 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:21.978132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33283 (local address 127.3.0.193:35205)
0928 21:49:21.978288 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:21.978335 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:21.979248 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:21.979489 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:21.979492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:21.979742 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:21.979943 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:21.979947 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:21.980679 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:21.980680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:21.981506 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:21.981509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:21.981883 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:21.981884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:21.981985 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:21.982441 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:21.982449 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:21.983885 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:21.985329 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:21.985335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:21.985337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:21.985559 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:21.985704 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:21.985704 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:21.985704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:21.985789 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:21.985962 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:21.985964 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:21.986077 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 21:49:21.986152 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:49:21.986265 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260928 21:49:21.986907 8833 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33283
I20260928 21:49:21.986989 8833 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:21.987129 8833 heartbeater.cc:511] Master 127.3.0.254:33283 requested a full tablet report, sending...
I20260928 21:49:21.987586 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 8704
I20260928 21:49:21.987636 8642 ts_manager.cc:194] Registered new tserver with Master: ca35a12106c74fbea52d84556468fc05 (127.3.0.193:42225)
I20260928 21:49:21.987685 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:21.987869 3075 external_mini_cluster.cc:1428] Setting key d5a2afa73ae6f6b9f4d70cc82a0ba5af
I20260928 21:49:21.988402 8642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:35205
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:49:22.008854 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33283
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:22.087759 8838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:22.087971 8838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:22.088006 8838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:22.089180 8838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:22.089221 8838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:22.089247 8838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:22.090656 8838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:22.090831 8838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:22.090996 8838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:22.095674 8843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:22.095675 8844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:22.095894 8846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:22.095894 8838 server_base.cc:1034] running on GCE node
I20260928 21:49:22.096156 8838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:22.096385 8838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:22.097530 8838 hybrid_clock.cc:648] HybridClock initialized: now 1790632162097523 us; error 38 us; skew 500 ppm
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:22.099826 8838 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:49:22.100277 8838 webserver.cc:460] Webserver started at http://127.3.0.194:42045/ using document root <none> and password file <none>
I20260928 21:49:22.100533 8838 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:22.100584 8838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:22.100701 8838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:22.101756 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "0f31c33c34884ac986006c3bf37514d7"
format_stamp: "Formatted at 2026-09-28 21:49:22 on dist-test-slave-jrck"
server_key: "7529386248b21ba4149b8c1fbe316f77"
server_key_iv: "89433abde924fd79a86fbc8b108421ad"
server_key_version: "encryptionkey@0"
I20260928 21:49:22.102077 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "0f31c33c34884ac986006c3bf37514d7"
format_stamp: "Formatted at 2026-09-28 21:49:22 on dist-test-slave-jrck"
server_key: "7529386248b21ba4149b8c1fbe316f77"
server_key_iv: "89433abde924fd79a86fbc8b108421ad"
server_key_version: "encryptionkey@0"
I20260928 21:49:22.103252 8838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:22.103883 8853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:22.104031 8838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:22.104110 8838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "0f31c33c34884ac986006c3bf37514d7"
format_stamp: "Formatted at 2026-09-28 21:49:22 on dist-test-slave-jrck"
server_key: "7529386248b21ba4149b8c1fbe316f77"
server_key_iv: "89433abde924fd79a86fbc8b108421ad"
server_key_version: "encryptionkey@0"
I20260928 21:49:22.104169 8838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:22.121789 8838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:22.123757 8838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:22.123891 8838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:22.124105 8838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:22.124383 8838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:22.124423 8838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:22.124504 8838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:22.124531 8838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:22.129760 8838 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:44049
I20260928 21:49:22.129814 8966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:44049 every 8 connection(s)
I20260928 21:49:22.130088 8838 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:49:22.132827 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 8838
I20260928 21:49:22.132925 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:22.133112 3075 external_mini_cluster.cc:1428] Setting key 5f0312486298318e3eb1a635941b455d
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:22.139211 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:22.130731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:55241 (local address 127.3.0.254:33283)
0928 21:49:22.130776 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:22.130778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:22.130976 (+ 198us) server_negotiation.cc:407] Connection header received
0928 21:49:22.131914 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:22.131916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:22.131930 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:22.131984 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:22.132610 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.133318 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.134446 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.134717 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.134786 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:22.136771 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:22.136773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:22.136773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:22.136781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:22.137998 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:22.138354 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:22.138355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:22.138356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:22.138407 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:22.138621 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:22.138622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:22.138622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:22.138724 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:22.138781 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:22.139035 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 21:49:22.139123 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:49:22.139359 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:22.130697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33283 (local address 127.3.0.194:55241)
0928 21:49:22.130863 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:22.130897 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:22.131806 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:22.132071 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:22.132075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:22.132312 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:22.132519 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:22.132523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.133438 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.133439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:22.134360 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:22.134363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.134785 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.134786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:22.134894 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:22.135341 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:22.135349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:22.136689 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:22.138070 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:22.138076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:22.138078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:22.138291 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:22.138466 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:22.138467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:22.138467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:22.138569 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:22.138777 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:22.138783 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:22.138921 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 21:49:22.139008 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:49:22.139153 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260928 21:49:22.139741 8967 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33283
I20260928 21:49:22.139832 8967 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:22.139997 8967 heartbeater.cc:511] Master 127.3.0.254:33283 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
I20260928 21:49:22.140370 8642 ts_manager.cc:194] Registered new tserver with Master: 0f31c33c34884ac986006c3bf37514d7 (127.3.0.194:44049)
I20260928 21:49:22.140762 8642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:55241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:49:22.154627 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33283
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:22.233543 8972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:22.233752 8972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:22.233774 8972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:22.234863 8972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:22.234897 8972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:22.234922 8972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:22.236366 8972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:22.236688 8972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:22.236907 8972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:22.241619 8977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:22.241564 8978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:22.241667 8972 server_base.cc:1034] running on GCE node
W20260928 21:49:22.241564 8980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:22.241966 8972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:22.242192 8972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:22.243325 8972 hybrid_clock.cc:648] HybridClock initialized: now 1790632162243312 us; error 28 us; skew 500 ppm
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:22.245707 8972 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:49:22.246132 8972 webserver.cc:460] Webserver started at http://127.3.0.195:34227/ using document root <none> and password file <none>
I20260928 21:49:22.246322 8972 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:22.246369 8972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:22.246492 8972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:22.247490 8972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "7670a2e480dc4da29a9b47cf43ade027"
format_stamp: "Formatted at 2026-09-28 21:49:22 on dist-test-slave-jrck"
server_key: "d2e4587c67572b57e9563529710b6539"
server_key_iv: "bf9cea529df90830baaa6180f4ae1527"
server_key_version: "encryptionkey@0"
I20260928 21:49:22.247815 8972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "7670a2e480dc4da29a9b47cf43ade027"
format_stamp: "Formatted at 2026-09-28 21:49:22 on dist-test-slave-jrck"
server_key: "d2e4587c67572b57e9563529710b6539"
server_key_iv: "bf9cea529df90830baaa6180f4ae1527"
server_key_version: "encryptionkey@0"
I20260928 21:49:22.249017 8972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:22.249650 8987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:22.249790 8972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:22.249868 8972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "7670a2e480dc4da29a9b47cf43ade027"
format_stamp: "Formatted at 2026-09-28 21:49:22 on dist-test-slave-jrck"
server_key: "d2e4587c67572b57e9563529710b6539"
server_key_iv: "bf9cea529df90830baaa6180f4ae1527"
server_key_version: "encryptionkey@0"
I20260928 21:49:22.249931 8972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:22.258435 8972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:22.260530 8972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:22.260675 8972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:22.260883 8972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:22.261196 8972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:22.261231 8972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:22.261258 8972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:22.261276 8972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:22.266700 8972 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:45041
I20260928 21:49:22.266752 9100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:45041 every 8 connection(s)
I20260928 21:49:22.267037 8972 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:49:22.268723 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 8972
I20260928 21:49:22.268810 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:22.269006 3075 external_mini_cluster.cc:1428] Setting key f8ce72564d7d017dc37c1f035b214f13
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:49:22.275406 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:22.267697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:59065 (local address 127.3.0.254:33283)
0928 21:49:22.267751 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:22.267752 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:22.267950 (+ 198us) server_negotiation.cc:407] Connection header received
0928 21:49:22.268916 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:22.268917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:22.268931 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:22.268998 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:22.269807 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.270324 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.271253 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.271427 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.271470 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:22.273410 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:22.273411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:22.273411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:22.273418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:22.274424 (+ 1006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:22.274738 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:22.274739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:22.274739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:22.274780 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:22.274950 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:22.274950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:22.274950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:22.275049 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:22.275094 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:22.275289 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 21:49:22.275344 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:49:22.275600 9103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:22.267682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33283 (local address 127.3.0.195:59065)
0928 21:49:22.267829 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:22.267865 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:22.268813 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:22.269069 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:22.269073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:22.269439 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:22.269710 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:22.269717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.270391 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.270392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:22.271193 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:22.271195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.271480 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.271480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:22.271575 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:22.272039 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:22.272047 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:22.273332 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:22.274486 (+ 1154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:22.274490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:22.274492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:22.274686 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:22.274822 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:22.274822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:22.274822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:22.274906 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:22.275090 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:22.275093 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:22.275198 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 21:49:22.275271 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:49:22.275382 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260928 21:49:22.275996 9101 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33283
I20260928 21:49:22.276085 9101 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:22.276248 9101 heartbeater.cc:511] Master 127.3.0.254:33283 requested a full tablet report, sending...
I20260928 21:49:22.276583 8642 ts_manager.cc:194] Registered new tserver with Master: 7670a2e480dc4da29a9b47cf43ade027 (127.3.0.195:45041)
I20260928 21:49:22.276928 8642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:59065
I20260928 21:49:22.280565 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/test-admin.keytab.
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:22.294050 3075 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 21:49:22.295487 9108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:49:22.294838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.3.0.193:42225)
0928 21:49:22.294985 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:22.294990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:22.295000 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:49:22.295025 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:22.295026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:22.295040 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:22.295102 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:22.295403 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57622: BlockingRecv error: recv got EOF from 127.0.0.1:57622 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
W20260928 21:49:22.295580 3075 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:22.296546 3075 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:22.297396 3075 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:49:22.304364 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:22.297865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.3.0.193:42225)
0928 21:49:22.297904 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:22.297906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:22.297915 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:49:22.297972 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:22.297973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:22.297981 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:22.298017 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:22.298354 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.299069 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.299728 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.300022 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.300068 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:22.301943 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:22.301952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:22.301952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:22.301959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:22.303381 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:22.303720 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:22.303723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:22.303724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:22.303787 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:22.303935 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:22.303935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:22.303936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:22.304069 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:22.304185 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:22.304218 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 21:49:22.304281 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 21:49:22.306058 9108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:49:22.305565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.3.0.193:42225)
0928 21:49:22.305603 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:22.305605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:22.305660 (+ 55us) server_negotiation.cc:407] Connection header received
0928 21:49:22.305717 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:22.305718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:22.305727 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:22.305761 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:22.306017 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57644: BlockingRecv error: recv got EOF from 127.0.0.1:57644 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 21:49:22.306143 3075 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:22.307094 3075 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:22.307875 3075 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:49:22.314081 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:22.308425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.3.0.193:42225)
0928 21:49:22.308494 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:22.308496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:22.308526 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:49:22.308578 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:22.308578 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:22.308586 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:22.308629 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:22.308958 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.309461 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.310140 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.310336 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.310406 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:22.312200 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:22.312201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:22.312201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:22.312207 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:22.313307 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:22.313584 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:22.313584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:22.313585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:22.313631 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:22.313752 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:22.313752 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:22.313753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:22.313830 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:22.313894 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:22.313965 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 21:49:22.314021 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 21:49:22.315451 9108 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:49:22.314969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.3.0.193:42225)
0928 21:49:22.315005 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:22.315007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:22.315036 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:49:22.315084 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:22.315085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:22.315092 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:22.315128 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:22.315406 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57660: BlockingRecv error: recv got EOF from 127.0.0.1:57660 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 21:49:22.315526 3075 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790632139349757-3075-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:22.316355 3075 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:22.317111 3075 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[8597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:49:22.323105 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:22.317607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.3.0.193:42225)
0928 21:49:22.317658 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:22.317660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:22.317682 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:49:22.317699 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:22.317700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:22.317708 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:22.317747 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:22.318063 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.318505 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.319144 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:22.319319 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:22.319369 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:22.321178 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:22.321179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:22.321179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:22.321186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:22.322192 (+ 1006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:22.322466 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:22.322467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:22.322467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:22.322516 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:22.322671 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:22.322672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:22.322673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:22.322760 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:22.322843 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:22.322936 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 21:49:22.323012 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:49:22.323715 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 8704
I20260928 21:49:22.327869 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 8838
I20260928 21:49:22.331434 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 8972
I20260928 21:49:22.334878 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 8610
2026-09-28T21:49:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1737 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[9121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[9121](info): set up 2 sockets
Sep 28 21:49:22 dist-test-slave-jrck krb5kdc[9121](info): commencing operation
krb5kdc: starting...
W20260928 21:49:23.288858 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:23 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.254.keytab.
Entry for principal oryx/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.254.keytab.
I20260928 21:49:23.397302 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:46579
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.254.keytab
--principal=oryx/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:46579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:23.474351 9134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:23.474568 9134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:23.474597 9134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:23.475766 9134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:23.475817 9134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:23.475836 9134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:23.475852 9134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:23.475867 9134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:23.477484 9134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:46579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.254.keytab
--principal=oryx/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:46579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:23.477762 9134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:23.477991 9134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:23.482961 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:23.482934 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:23.482928 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:23.483223 9134 server_base.cc:1034] running on GCE node
I20260928 21:49:23.483448 9134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:23.483724 9134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:23.484856 9134 hybrid_clock.cc:648] HybridClock initialized: now 1790632163484840 us; error 28 us; skew 500 ppm
Sep 28 21:49:23 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:23.487196 9134 init.cc:377] Logged in from keytab as oryx/127.3.0.254@KRBTEST.COM (short username oryx)
I20260928 21:49:23.487618 9134 webserver.cc:460] Webserver started at http://127.3.0.254:41921/ using document root <none> and password file <none>
I20260928 21:49:23.487826 9134 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:23.487874 9134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:23.488001 9134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:23.489058 9134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "06fcd49e1d8745739eb7a281ab96ea8e"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "0baf053c54006bf3af3cf09ef9f59a9f"
server_key_iv: "939c18ae79bc3bc180a459c55c1b6f8b"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.489367 9134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "06fcd49e1d8745739eb7a281ab96ea8e"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "0baf053c54006bf3af3cf09ef9f59a9f"
server_key_iv: "939c18ae79bc3bc180a459c55c1b6f8b"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.490540 9134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:23.491207 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:23.491346 9134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:23.491425 9134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "06fcd49e1d8745739eb7a281ab96ea8e"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "0baf053c54006bf3af3cf09ef9f59a9f"
server_key_iv: "939c18ae79bc3bc180a459c55c1b6f8b"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.491482 9134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:23.501448 9134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:23.503402 9134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:23.503532 9134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:23.507198 9134 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:46579
I20260928 21:49:23.507227 9201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:46579 every 8 connection(s)
I20260928 21:49:23.507521 9134 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:23.508399 9202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:23.510545 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e: Bootstrap starting.
I20260928 21:49:23.511073 9202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:23.511313 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 9134
I20260928 21:49:23.511322 9202 log.cc:824] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:23.511415 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:23.511610 3075 external_mini_cluster.cc:1428] Setting key 21852f167e2a41d98516dab4d3dfb0b5
I20260928 21:49:23.512046 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e: No bootstrap required, opened a new log
I20260928 21:49:23.513837 9202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } }
I20260928 21:49:23.514012 9202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:23.514081 9202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06fcd49e1d8745739eb7a281ab96ea8e, State: Initialized, Role: FOLLOWER
I20260928 21:49:23.514210 9202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } }
I20260928 21:49:23.514286 9202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:23.514317 9202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:23.514364 9202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:23.515342 9202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } }
I20260928 21:49:23.515447 9202 leader_election.cc:302] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06fcd49e1d8745739eb7a281ab96ea8e; no voters:
I20260928 21:49:23.515588 9202 leader_election.cc:288] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:23.515614 9207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:23.515761 9207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 1 LEADER]: Becoming Leader. State: Replica: 06fcd49e1d8745739eb7a281ab96ea8e, State: Running, Role: LEADER
I20260928 21:49:23.515813 9202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:23.515883 9207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } }
I20260928 21:49:23.516247 9208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } }
I20260928 21:49:23.516320 9208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:23.516518 9209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06fcd49e1d8745739eb7a281ab96ea8e. Latest consensus state: current_term: 1 leader_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } }
I20260928 21:49:23.516636 9209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:23.516788 9213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:23.517272 9213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:23.518772 9213 catalog_manager.cc:1347] Generated new cluster ID: 1fac966ae3204ced96d738c1f76e5f5b
I20260928 21:49:23.518819 9213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:49:23 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:23.524094 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:23.512089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.3.0.254:46579)
0928 21:49:23.512272 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:23.512283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:23.512315 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:49:23.513809 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:23.513817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:23.513881 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:23.514035 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:23.514686 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.516043 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.517105 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.517664 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.517768 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:23.520341 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:23.520346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:23.520346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:23.520357 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:23.522137 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:23.522511 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:23.522514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:23.522514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:23.523037 (+ 523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:23.523238 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:23.523239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:23.523239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:23.523527 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:23.523668 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:23.523772 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 21:49:23.523892 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260928 21:49:23.545892 9213 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:23.546617 9213 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:23.550158 9213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e: Generated new TSK 0
I20260928 21:49:23.550370 9213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab.
Entry for principal oryx/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab.
I20260928 21:49:23.576941 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46579
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab
--principal=oryx/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:23.654495 9228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:23.654676 9228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:23.654697 9228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:23.655741 9228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:23.655772 9228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:23.655797 9228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:23.657207 9228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab
--principal=oryx/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:23.657383 9228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:23.657549 9228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:23.662114 9234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:23.662106 9236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:23.662106 9233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:23.662223 9228 server_base.cc:1034] running on GCE node
I20260928 21:49:23.662396 9228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:23.662633 9228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:23.663765 9228 hybrid_clock.cc:648] HybridClock initialized: now 1790632163663755 us; error 28 us; skew 500 ppm
Sep 28 21:49:23 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:23.666093 9228 init.cc:377] Logged in from keytab as oryx/127.3.0.193@KRBTEST.COM (short username oryx)
I20260928 21:49:23.666477 9228 webserver.cc:460] Webserver started at http://127.3.0.193:41685/ using document root <none> and password file <none>
I20260928 21:49:23.666661 9228 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:23.666697 9228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:23.666783 9228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:23.667759 9228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "645f2026f84d4693afbf688a5961bd79"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "daad60fb030093ceb365a258f3e17acc"
server_key_iv: "21b516939694920314e2116e9ddcc0cb"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.668040 9228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "645f2026f84d4693afbf688a5961bd79"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "daad60fb030093ceb365a258f3e17acc"
server_key_iv: "21b516939694920314e2116e9ddcc0cb"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.669227 9228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:49:23.669822 9243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:23.669970 9228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:23.670048 9228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "645f2026f84d4693afbf688a5961bd79"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "daad60fb030093ceb365a258f3e17acc"
server_key_iv: "21b516939694920314e2116e9ddcc0cb"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.670101 9228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:23.684185 9228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:23.686178 9228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:23.686290 9228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:23.686460 9228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:23.686699 9228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:23.686728 9228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:23.686748 9228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:23.686764 9228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:23.691875 9228 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:38505
I20260928 21:49:23.691947 9356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:38505 every 8 connection(s)
I20260928 21:49:23.692196 9228 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:49:23 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:23.700269 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:23.692841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:51233 (local address 127.3.0.254:46579)
0928 21:49:23.692892 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:23.692894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:23.693096 (+ 202us) server_negotiation.cc:407] Connection header received
0928 21:49:23.693589 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:23.693589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:23.693601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:23.693656 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:23.694519 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.695039 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.696006 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.696213 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.696269 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:23.698099 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:23.698100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:23.698100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:23.698107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:23.699228 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:23.699557 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:23.699558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:23.699558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:23.699603 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:23.699769 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:23.699769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:23.699770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:23.699861 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:23.699903 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:23.700109 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 21:49:23.700171 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:49:23.700397 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:23.692815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:51233)
0928 21:49:23.692981 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:23.693022 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:23.693499 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:23.693710 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:23.693713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:23.694261 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:23.694446 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:23.694451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.695115 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.695115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:23.695911 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:23.695913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.696274 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.696274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:23.696364 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:23.696755 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:23.696763 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:23.698015 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:23.699291 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:23.699297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:23.699299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:23.699503 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:23.699645 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:23.699646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:23.699646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:23.699730 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:23.699900 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:23.699902 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:23.700016 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 21:49:23.700089 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:49:23.700202 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260928 21:49:23.700870 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 9228
I20260928 21:49:23.700892 9357 heartbeater.cc:347] Connected to a master server at 127.3.0.254:46579
I20260928 21:49:23.700961 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:23.700989 9357 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:23.701169 3075 external_mini_cluster.cc:1428] Setting key f0874ad1292ab9e4994f8872d9cb50e6
I20260928 21:49:23.701169 9357 heartbeater.cc:511] Master 127.3.0.254:46579 requested a full tablet report, sending...
I20260928 21:49:23.701648 9166 ts_manager.cc:194] Registered new tserver with Master: 645f2026f84d4693afbf688a5961bd79 (127.3.0.193:38505)
I20260928 21:49:23.702262 9166 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:51233
WARNING: no policy specified for oryx/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab.
Entry for principal oryx/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab.
I20260928 21:49:23.721285 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46579
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab
--principal=oryx/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:23.798772 9362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:23.798974 9362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:23.799005 9362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:23.800148 9362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:23.800189 9362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:23.800213 9362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:23.801749 9362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab
--principal=oryx/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:23.801991 9362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:23.802206 9362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:23.806854 9367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:23.806857 9370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:23.806857 9368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:23.806900 9362 server_base.cc:1034] running on GCE node
I20260928 21:49:23.807235 9362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:23.807451 9362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:23.808599 9362 hybrid_clock.cc:648] HybridClock initialized: now 1790632163808583 us; error 28 us; skew 500 ppm
Sep 28 21:49:23 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:23.810896 9362 init.cc:377] Logged in from keytab as oryx/127.3.0.194@KRBTEST.COM (short username oryx)
I20260928 21:49:23.811311 9362 webserver.cc:460] Webserver started at http://127.3.0.194:43779/ using document root <none> and password file <none>
I20260928 21:49:23.811532 9362 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:23.811581 9362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:23.811704 9362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:23.812783 9362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "6f2ec22333cb388e93d3291a83cd2d08"
server_key_iv: "a7ede8c5063673ce830307bd0e7091af"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.813103 9362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "6f2ec22333cb388e93d3291a83cd2d08"
server_key_iv: "a7ede8c5063673ce830307bd0e7091af"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.814301 9362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:23.814944 9377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:23.815088 9362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:23.815165 9362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "6f2ec22333cb388e93d3291a83cd2d08"
server_key_iv: "a7ede8c5063673ce830307bd0e7091af"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.815227 9362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:23.832149 9362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:23.833811 9362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:23.833940 9362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:23.834131 9362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:23.834404 9362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:23.834436 9362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:23.834465 9362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:23.834497 9362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:23.839658 9362 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:35407
I20260928 21:49:23.839716 9490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:35407 every 8 connection(s)
I20260928 21:49:23.839996 9362 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:49:23.845124 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 9362
I20260928 21:49:23.845211 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
Sep 28 21:49:23 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:23.845394 3075 external_mini_cluster.cc:1428] Setting key 4504e80919e112a4b9f90330a9e70722
I20260928 21:49:23.848348 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:23.840647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:55379 (local address 127.3.0.254:46579)
0928 21:49:23.840722 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:23.840725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:23.840869 (+ 144us) server_negotiation.cc:407] Connection header received
0928 21:49:23.841304 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:23.841305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:23.841314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:23.841362 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:23.842203 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.842681 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.843585 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.843758 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.843800 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:23.845830 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:23.845832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:23.845832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:23.845838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:23.847125 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:23.847507 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:23.847508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:23.847509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:23.847563 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:23.847766 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:23.847767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:23.847767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:23.847876 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:23.847931 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:23.848176 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 21:49:23.848261 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:49:23.848516 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:23.840619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:55379)
0928 21:49:23.840754 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:23.840789 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:23.841231 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:23.841415 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:23.841418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:23.841951 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:23.842138 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:23.842142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.842740 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.842741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:23.843524 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:23.843526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.843809 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.843810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:23.843904 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:23.844353 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:23.844363 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:23.845753 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:23.847194 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:23.847199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:23.847201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:23.847432 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:23.847611 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:23.847612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:23.847612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:23.847713 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:23.847935 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:23.847941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:23.848066 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 21:49:23.848149 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 21:49:23.848291 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260928 21:49:23.848932 9491 heartbeater.cc:347] Connected to a master server at 127.3.0.254:46579
I20260928 21:49:23.849005 9491 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:23.849164 9491 heartbeater.cc:511] Master 127.3.0.254:46579 requested a full tablet report, sending...
I20260928 21:49:23.849515 9166 ts_manager.cc:194] Registered new tserver with Master: 7f84f2e6c1ea499aabc1b2c6cf406f11 (127.3.0.194:35407)
I20260928 21:49:23.849861 9166 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:55379
WARNING: no policy specified for oryx/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab.
Entry for principal oryx/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab.
I20260928 21:49:23.867092 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46579
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab
--principal=oryx/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:23.944351 9496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:23.944569 9496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:23.944592 9496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:23.945760 9496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:23.945803 9496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:23.945829 9496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:23.947249 9496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab
--principal=oryx/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:46579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:23.947467 9496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:23.947667 9496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:23.952234 9502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:23.952229 9501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:23.952313 9496 server_base.cc:1034] running on GCE node
W20260928 21:49:23.952229 9504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:23.952616 9496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:23.952819 9496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:23.953970 9496 hybrid_clock.cc:648] HybridClock initialized: now 1790632163953953 us; error 31 us; skew 500 ppm
Sep 28 21:49:23 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:23.956218 9496 init.cc:377] Logged in from keytab as oryx/127.3.0.195@KRBTEST.COM (short username oryx)
I20260928 21:49:23.956669 9496 webserver.cc:460] Webserver started at http://127.3.0.195:43805/ using document root <none> and password file <none>
I20260928 21:49:23.956894 9496 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:23.956941 9496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:23.957054 9496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:23.958050 9496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "d046de81da1246bfb25cf3bbe0a92eb9"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "08d86712f45cab0ed9971cfedca4397a"
server_key_iv: "748823baf69addb8f394845b9d63428b"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.958370 9496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "d046de81da1246bfb25cf3bbe0a92eb9"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "08d86712f45cab0ed9971cfedca4397a"
server_key_iv: "748823baf69addb8f394845b9d63428b"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.959522 9496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:23.960114 9511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:23.960256 9496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:23.960346 9496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "d046de81da1246bfb25cf3bbe0a92eb9"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "08d86712f45cab0ed9971cfedca4397a"
server_key_iv: "748823baf69addb8f394845b9d63428b"
server_key_version: "encryptionkey@0"
I20260928 21:49:23.960412 9496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:23.983649 9496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:23.985414 9496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:23.985553 9496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:23.985752 9496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:23.986152 9496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:23.986194 9496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:23.986224 9496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:23.986251 9496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:23.991600 9496 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:42211
I20260928 21:49:23.991641 9624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:42211 every 8 connection(s)
I20260928 21:49:23.991937 9496 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:49:23 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:24.000041 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:23.992585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:60009 (local address 127.3.0.254:46579)
0928 21:49:23.992648 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:23.992650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:23.992828 (+ 178us) server_negotiation.cc:407] Connection header received
0928 21:49:23.993288 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:23.993289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:23.993299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:23.993333 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:23.994218 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.994752 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.995775 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.995956 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.995999 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:23.997816 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:23.997818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:23.997818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:23.997825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:23.999037 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:23.999364 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:23.999365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:23.999365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:23.999410 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:23.999590 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:23.999591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:23.999591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:23.999682 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:23.999724 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:23.999919 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 21:49:23.999977 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:49:24.000207 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:23.992576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:60009)
0928 21:49:23.992720 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:23.992755 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:23.993202 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:23.993392 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:23.993396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:23.993959 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:23.994145 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:23.994150 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.994824 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.994824 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:23.995709 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:23.995712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:23.996008 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:23.996009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:23.996102 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:23.996521 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:23.996529 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:23.997731 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:23.999108 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:23.999114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:23.999115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:23.999309 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:23.999464 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:23.999465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:23.999465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:23.999550 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:23.999722 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:23.999725 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:23.999828 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 21:49:23.999901 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.000012 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260928 21:49:24.000669 9625 heartbeater.cc:347] Connected to a master server at 127.3.0.254:46579
I20260928 21:49:24.000754 9625 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:24.000931 9625 heartbeater.cc:511] Master 127.3.0.254:46579 requested a full tablet report, sending...
I20260928 21:49:24.001267 9166 ts_manager.cc:194] Registered new tserver with Master: d046de81da1246bfb25cf3bbe0a92eb9 (127.3.0.195:42211)
I20260928 21:49:24.001295 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 9496
I20260928 21:49:24.001386 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:24.001564 3075 external_mini_cluster.cc:1428] Setting key 22f24d38de768124f3bd36d4f68e1350
I20260928 21:49:24.001663 9166 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:60009
I20260928 21:49:24.002326 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.0.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.253.keytab.
Entry for principal oryx/127.3.0.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.253.keytab.
I20260928 21:49:24.024050 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.253:38339
--webserver_interface=127.3.0.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.253.keytab
--principal=oryx/127.3.0.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:46579,127.3.0.253:38339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:24.103197 9630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:24.103384 9630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:24.103403 9630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:24.104524 9630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:24.104562 9630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:24.104575 9630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:24.104588 9630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:24.104627 9630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:24.106057 9630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:46579,127.3.0.253:38339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.253.keytab
--principal=oryx/127.3.0.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.253:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.0.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
W20260928 21:49:24.106153 9630 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.0.254:46579,127.3.0.253:38339'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 21:49:24.106297 9630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:24.106492 9630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:24.110782 9635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.110812 9638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:24.110832 9630 server_base.cc:1034] running on GCE node
W20260928 21:49:24.110790 9636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:24.111145 9630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:24.111428 9630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:24.112517 9630 hybrid_clock.cc:648] HybridClock initialized: now 1790632164112499 us; error 26 us; skew 500 ppm
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:24.114675 9630 init.cc:377] Logged in from keytab as oryx/127.3.0.253@KRBTEST.COM (short username oryx)
I20260928 21:49:24.115034 9630 webserver.cc:460] Webserver started at http://127.3.0.253:42765/ using document root <none> and password file <none>
I20260928 21:49:24.115216 9630 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:24.115258 9630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:24.115384 9630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:24.116477 9630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data/instance:
uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587"
format_stamp: "Formatted at 2026-09-28 21:49:24 on dist-test-slave-jrck"
server_key: "d635fdcb7d86e19f1e0e28f4bcd6b4d3"
server_key_iv: "3ac633ad82e3be0ab1bad6cb46d67df6"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.116811 9630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal/instance:
uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587"
format_stamp: "Formatted at 2026-09-28 21:49:24 on dist-test-slave-jrck"
server_key: "d635fdcb7d86e19f1e0e28f4bcd6b4d3"
server_key_iv: "3ac633ad82e3be0ab1bad6cb46d67df6"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.118086 9630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:24.118705 9645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.118835 9630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:24.118911 9630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal
uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587"
format_stamp: "Formatted at 2026-09-28 21:49:24 on dist-test-slave-jrck"
server_key: "d635fdcb7d86e19f1e0e28f4bcd6b4d3"
server_key_iv: "3ac633ad82e3be0ab1bad6cb46d67df6"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.118978 9630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:24.137631 9630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:24.139364 9630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:24.139498 9630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:24.143052 9630 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.253:38339
I20260928 21:49:24.143083 9697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.253:38339 every 8 connection(s)
I20260928 21:49:24.143373 9630 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
I20260928 21:49:24.144227 9698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:24.145670 9698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:49:24.148343 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 9630
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.253@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.0.253@KRBTEST.COM
I20260928 21:49:24.154189 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.146307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.3.0.254:46579)
0928 21:49:24.146367 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.146369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.146534 (+ 165us) server_negotiation.cc:407] Connection header received
0928 21:49:24.146984 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.146986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.146998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.147049 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.147608 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.148098 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.149328 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.149627 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.149701 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.151669 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.151670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.151671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.151678 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.152984 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.153318 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.153319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.153319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.153378 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.153578 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.153579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.153579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.153760 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.153831 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.154069 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.154129 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:49:24.154402 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.146282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:59510)
0928 21:49:24.146429 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.146461 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.146884 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.147115 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.147118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.147339 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.147524 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.147528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.148176 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.148177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.149253 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.149255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.149701 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.149702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.149795 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.150231 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.150240 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.151594 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.153058 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.153063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.153065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.153255 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.153431 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.153432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.153432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.153523 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.153830 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.153836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.153947 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.154041 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.154209 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260928 21:49:24.154953 9698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:49:24.156675 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.148775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.3.0.253:38339)
0928 21:49:24.149093 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.149097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.149115 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:49:24.149135 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.149137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.149159 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.149232 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.149827 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.150775 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.151558 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.151961 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.152020 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.154037 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.154042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.154042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.154048 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.155659 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.155981 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.155983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.155983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.156025 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.156180 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.156181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.156181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.156285 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.156437 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.156514 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.156581 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":83,"threads_started":1}
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.253@KRBTEST.COM for oryx/127.3.0.253@KRBTEST.COM
I20260928 21:49:24.161186 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.155158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:38339 (local address 127.0.0.1:47948)
0928 21:49:24.155194 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.155203 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.155235 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.155385 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.155386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.155556 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.155593 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.155594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.156357 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.156358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.157048 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.157050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.157419 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.157420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.157469 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.157730 (+ 261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.157733 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.158701 (+ 968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.160477 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.160478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.160479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.160658 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.160807 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.160807 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.160808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.160851 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.160999 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.161000 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.161028 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.161069 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.161132 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 21:49:24.161293 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.155204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.3.0.253:38339)
0928 21:49:24.155307 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.155308 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.155312 (+ 4us) server_negotiation.cc:407] Connection header received
0928 21:49:24.155323 (+ 11us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.155323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.155329 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.155375 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.155651 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.156292 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.157117 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.157365 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.157413 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.158766 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.158767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.158767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.158772 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.160383 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.160719 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.160720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.160720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.160765 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.160886 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.160886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.160887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.160966 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.161005 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.161084 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.161174 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":90,"thread_start_us":71,"threads_started":1}
I20260928 21:49:24.162395 9698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: Bootstrap starting.
I20260928 21:49:24.162981 9698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:24.163211 9698 log.cc:824] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:24.163846 9698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: No bootstrap required, opened a new log
I20260928 21:49:24.165199 9698 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } }
I20260928 21:49:24.165307 9698 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:24.165328 9698 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2baa60ebbbf48aa8aa6636e8a5f6587, State: Initialized, Role: FOLLOWER
I20260928 21:49:24.165441 9698 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } }
I20260928 21:49:24.165704 9698 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:24.165669 9705 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } } }
I20260928 21:49:24.165759 9705 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:49:24.167055 9716 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:49:24.167107 9716 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:49:24.168660 9630 master_runner.cc:428] Detected that this master f2baa60ebbbf48aa8aa6636e8a5f6587 is joining an existing cluster
I20260928 21:49:24.168706 9630 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.0.253:38339
I20260928 21:49:24.178814 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.169619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:38339 (local address 127.0.0.1:47956)
0928 21:49:24.172577 (+ 2958us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.172587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.172642 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.172860 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.172861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.173066 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.173126 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.173128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.174012 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.174013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.174864 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.174866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.175215 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.175216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.175266 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.175649 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.175654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.175961 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.177762 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.177763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.177764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.177954 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.178129 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.178130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.178130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.178179 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.178432 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.178433 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.178496 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.178569 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.178665 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2934,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260928 21:49:24.178875 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.169745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.3.0.253:38339)
0928 21:49:24.169781 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.169783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.172641 (+ 2858us) server_negotiation.cc:407] Connection header received
0928 21:49:24.172692 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.172693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.172702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.172759 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.173191 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.173915 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.174947 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.175142 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.175192 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.176032 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.176033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.176033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.176040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.177674 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.178018 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.178019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.178020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.178069 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.178226 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.178228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.178228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.178359 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.178432 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.178594 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.178724 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":18}
I20260928 21:49:24.179960 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.169703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.3.0.254:46579)
0928 21:49:24.169761 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.169763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.172648 (+ 2885us) server_negotiation.cc:407] Connection header received
0928 21:49:24.172766 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.172767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.172778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.172826 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.173224 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.173976 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.175862 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.176137 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.176190 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.177261 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.177263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.177263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.177269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.178769 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.179303 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.179304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.179305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.179360 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.179522 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.179522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.179523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.179626 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.179679 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.179812 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.179876 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:49:24.179965 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.169619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:59516)
0928 21:49:24.172578 (+ 2959us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.172590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.172674 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.172843 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.172844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.173102 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.173160 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.173162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.174952 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.174953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.175780 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.175782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.176195 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.176196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.176265 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.176833 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.176837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.177181 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.178846 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.178846 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.178847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.179217 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.179417 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.179417 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.179418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.179471 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.179677 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.179678 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.179716 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.179776 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.179866 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2927,"thread_start_us":75,"threads_started":1}
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.254@KRBTEST.COM for oryx/127.3.0.253@KRBTEST.COM
I20260928 21:49:24.188653 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.182234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.3.0.253:38339)
0928 21:49:24.182301 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.182303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.182358 (+ 55us) server_negotiation.cc:407] Connection header received
0928 21:49:24.182425 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.182426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.182435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.182491 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.182827 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.183523 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.184235 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.184554 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.184614 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.186505 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.186506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.186506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.186512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.187653 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.188039 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.188039 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.188040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.188082 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.188230 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.188231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.188231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.188311 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.188352 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.188551 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.188601 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:49:24.188750 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.182155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:38339 (local address 127.0.0.1:47964)
0928 21:49:24.182271 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.182314 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.182374 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.182496 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.182498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.182726 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.182769 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.182771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.183603 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.183604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.184169 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.184170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.184613 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.184614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.184667 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.185048 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.185052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.186394 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.187721 (+ 1327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.187727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.187730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.187976 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.188121 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.188122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.188122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.188180 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.188360 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.188365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.188424 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.188533 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.188638 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
I20260928 21:49:24.189257 9166 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.0.253:38339: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587"
member_type: NON_VOTER
last_known_addr {
host: "127.3.0.253"
port: 38339
}
attrs {
promote: true
}
}
dest_uuid: "06fcd49e1d8745739eb7a281ab96ea8e"
cas_config_opid_index: -1
I20260928 21:49:24.189535 9166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } }
I20260928 21:49:24.189927 9726 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f2baa60ebbbf48aa8aa6636e8a5f6587 (127.3.0.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } } }
I20260928 21:49:24.190253 9672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:24.190999 9726 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.0.253:38339
I20260928 21:49:24.191090 9725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } } }
I20260928 21:49:24.191179 9725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:24.191460 9672 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Refusing update from remote peer 06fcd49e1d8745739eb7a281ab96ea8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 21:49:24.191731 9630 master.cc:537] Master@127.3.0.253:38339 shutting down...
I20260928 21:49:24.191736 9725 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:49:24.192487 9151 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 21:49:24.192886 9151 consensus_peers.cc:487] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e -> Peer f2baa60ebbbf48aa8aa6636e8a5f6587 (127.3.0.253:38339): Couldn't send request to peer f2baa60ebbbf48aa8aa6636e8a5f6587. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:49:24.194725 9630 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:24.194808 9630 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:24.194869 9630 tablet_replica.cc:330] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: stopping tablet replica
I20260928 21:49:24.206975 9630 master.cc:552] Master@127.3.0.253:38339 shutdown complete.
I20260928 21:49:24.207013 9630 master_runner.cc:305] Clearing existing system tablet
I20260928 21:49:24.207120 9630 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:24.207159 9630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:24.207757 9630 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:24.208287 9730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.208397 9630 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:24.208444 9630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.208506 9630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal
uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587"
format_stamp: "Formatted at 2026-09-28 21:49:24 on dist-test-slave-jrck"
server_key: "d635fdcb7d86e19f1e0e28f4bcd6b4d3"
server_key_iv: "3ac633ad82e3be0ab1bad6cb46d67df6"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.209065 9630 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:24.210481 9630 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:49:24.210532 9630 log.cc:1196] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 21:49:24.210685 9630 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: Deleting consensus metadata
I20260928 21:49:24.210777 9630 master_runner.cc:315] Copying system tablet from 127.3.0.254:46579
I20260928 21:49:24.211292 9630 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: tablet copy: Beginning tablet copy session from remote peer at address 127.3.0.254:46579
I20260928 21:49:24.216713 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.211568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:59528)
0928 21:49:24.211696 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.211705 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.211762 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.211874 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.211874 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.212029 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.212082 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.212084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.212755 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.212756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.213292 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.213293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.213642 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.213643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.213700 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.214146 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.214150 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.214413 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.215731 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.215732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.215733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.215891 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.216066 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.216066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.216067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.216121 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.216353 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.216354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.216397 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.216491 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.216584 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260928 21:49:24.216727 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.211609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.3.0.254:46579)
0928 21:49:24.211650 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.211652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.211776 (+ 124us) server_negotiation.cc:407] Connection header received
0928 21:49:24.211808 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.211809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.211818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.211869 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.212151 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.212693 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.213364 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.213585 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.213638 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.214511 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.214512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.214513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.214519 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.215672 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.215961 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.215962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.215962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.216016 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.216184 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.216185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.216185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.216301 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.216356 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.216578 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.216647 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:49:24.216971 9186 tablet_copy_service.cc:140] P 06fcd49e1d8745739eb7a281ab96ea8e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f2baa60ebbbf48aa8aa6636e8a5f6587 ({username='oryx', principal='oryx/127.3.0.253@KRBTEST.COM'} at 127.0.0.1:59528)
I20260928 21:49:24.217048 9186 tablet_copy_service.cc:161] P 06fcd49e1d8745739eb7a281ab96ea8e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f2baa60ebbbf48aa8aa6636e8a5f6587 at {username='oryx', principal='oryx/127.3.0.253@KRBTEST.COM'} at 127.0.0.1:59528: session id = f2baa60ebbbf48aa8aa6636e8a5f6587-00000000000000000000000000000000
I20260928 21:49:24.217694 9186 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:49:24.218214 9630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:49:24.219156 9630 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: tablet copy: Starting download of 0 data blocks...
I20260928 21:49:24.219269 9630 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: tablet copy: Starting download of 1 WAL segments...
I20260928 21:49:24.219993 9630 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:49:24.220743 9186 tablet_copy_service.cc:342] P 06fcd49e1d8745739eb7a281ab96ea8e: Request end of tablet copy session f2baa60ebbbf48aa8aa6636e8a5f6587-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.0.253@KRBTEST.COM'} at 127.0.0.1:59528
I20260928 21:49:24.220793 9186 tablet_copy_service.cc:433] P 06fcd49e1d8745739eb7a281ab96ea8e: ending tablet copy session f2baa60ebbbf48aa8aa6636e8a5f6587-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f2baa60ebbbf48aa8aa6636e8a5f6587
I20260928 21:49:24.223126 9630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:24.224112 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.224228 9737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.224215 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:24.224414 9630 server_base.cc:1034] running on GCE node
I20260928 21:49:24.224539 9630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:24.224670 9630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:24.225775 9630 hybrid_clock.cc:648] HybridClock initialized: now 1790632164225761 us; error 33 us; skew 500 ppm
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:24.227087 9630 init.cc:377] Logged in from keytab as oryx/127.3.0.253@KRBTEST.COM (short username oryx)
I20260928 21:49:24.227331 9630 webserver.cc:460] Webserver started at http://127.3.0.253:35627/ using document root <none> and password file <none>
I20260928 21:49:24.227437 9630 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:24.227468 9630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:24.228087 9630 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:24.228592 9747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.228706 9630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.228742 9630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal
uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587"
format_stamp: "Formatted at 2026-09-28 21:49:24 on dist-test-slave-jrck"
server_key: "d635fdcb7d86e19f1e0e28f4bcd6b4d3"
server_key_iv: "3ac633ad82e3be0ab1bad6cb46d67df6"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.228792 9630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:24.245437 9630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:24.246901 9630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:24.249142 9630 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.253:38339
I20260928 21:49:24.249181 9799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.253:38339 every 8 connection(s)
I20260928 21:49:24.249402 9630 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
I20260928 21:49:24.250187 9800 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:49:24.250695 9800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: Bootstrap starting.
I20260928 21:49:24.253400 9800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:49:24.253602 9800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: Bootstrap complete.
I20260928 21:49:24.253772 9800 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } }
I20260928 21:49:24.253892 9800 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f2baa60ebbbf48aa8aa6636e8a5f6587, State: Initialized, Role: LEARNER
I20260928 21:49:24.253937 9800 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } }
I20260928 21:49:24.254118 9800 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:24.254106 9803 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } } }
I20260928 21:49:24.254205 9803 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:49:24.255511 9814 catalog_manager.cc:1259] Loaded cluster ID: 1fac966ae3204ced96d738c1f76e5f5b
I20260928 21:49:24.255555 9814 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: loading cluster ID for follower catalog manager: success
I20260928 21:49:24.256444 9814 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: acquiring CA information for follower catalog manager: success
I20260928 21:49:24.256696 9814 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:49:24.263864 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.258059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.3.0.253:38339)
0928 21:49:24.258161 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.258163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.258170 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:49:24.258191 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.258191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.258199 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.258258 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.258742 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.259428 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.260120 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.260341 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.260401 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.261545 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.261546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.261546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.261552 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.262845 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.263180 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.263181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.263182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.263233 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.263389 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.263390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.263390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.263484 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.263567 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.263690 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.263762 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
WARNING: no policy specified for oryx/127.3.0.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.252.keytab.
Entry for principal oryx/127.3.0.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.252.keytab.
I20260928 21:49:24.285375 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.252:39847
--webserver_interface=127.3.0.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.252.keytab
--principal=oryx/127.3.0.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:46579,127.3.0.253:38339,127.3.0.252:39847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:24.366945 9818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:24.367139 9818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:24.367161 9818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:24.368331 9818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:24.368371 9818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:24.368383 9818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:24.368407 9818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:24.368429 9818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:24.369951 9818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:46579,127.3.0.253:38339,127.3.0.252:39847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.252.keytab
--principal=oryx/127.3.0.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.252:39847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.0.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:24.370173 9818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:24.370395 9818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:24.374841 9824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.374842 9826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.374879 9823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:24.374950 9818 server_base.cc:1034] running on GCE node
I20260928 21:49:24.375227 9818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:24.375480 9818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:24.376613 9818 hybrid_clock.cc:648] HybridClock initialized: now 1790632164376597 us; error 27 us; skew 500 ppm
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:24.378842 9818 init.cc:377] Logged in from keytab as oryx/127.3.0.252@KRBTEST.COM (short username oryx)
I20260928 21:49:24.379254 9818 webserver.cc:460] Webserver started at http://127.3.0.252:40661/ using document root <none> and password file <none>
I20260928 21:49:24.379460 9818 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:24.379508 9818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:24.379621 9818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:24.380664 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data/instance:
uuid: "73e4afeb2e034cc1be7c21e90c025570"
format_stamp: "Formatted at 2026-09-28 21:49:24 on dist-test-slave-jrck"
server_key: "6b7aeae5da933a1af68b2e34c9c95d40"
server_key_iv: "24f0e946df2c6698c3f038793e37c10e"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.380967 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal/instance:
uuid: "73e4afeb2e034cc1be7c21e90c025570"
format_stamp: "Formatted at 2026-09-28 21:49:24 on dist-test-slave-jrck"
server_key: "6b7aeae5da933a1af68b2e34c9c95d40"
server_key_iv: "24f0e946df2c6698c3f038793e37c10e"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.382114 9818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:24.383476 9833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.383628 9818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:24.383709 9818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal
uuid: "73e4afeb2e034cc1be7c21e90c025570"
format_stamp: "Formatted at 2026-09-28 21:49:24 on dist-test-slave-jrck"
server_key: "6b7aeae5da933a1af68b2e34c9c95d40"
server_key_iv: "24f0e946df2c6698c3f038793e37c10e"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.383776 9818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:24.401739 9818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:24.403403 9818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:24.403533 9818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:24.407958 9818 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.252:39847
I20260928 21:49:24.407999 9885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.252:39847 every 8 connection(s)
I20260928 21:49:24.408318 9818 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
I20260928 21:49:24.409240 9886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:24.409672 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 9818
I20260928 21:49:24.410874 9886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.252@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.0.252@KRBTEST.COM
I20260928 21:49:24.418486 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.411585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.3.0.254:46579)
0928 21:49:24.411630 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.411632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.411706 (+ 74us) server_negotiation.cc:407] Connection header received
0928 21:49:24.411769 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.411770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.411780 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.411832 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.412281 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.412824 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.413602 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.413792 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.413846 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.415643 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.415644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.415644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.415651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.416980 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.417460 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.417461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.417461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.417518 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.417770 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.417771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.417771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.417896 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.417952 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.418317 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.418394 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:24.419011 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.411405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:59542)
0928 21:49:24.411613 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.411650 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.411711 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.411848 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.411851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.412105 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.412199 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.412204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.412917 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.412919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.413527 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.413529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.413855 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.413856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.413920 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.414281 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.414289 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.415566 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.417051 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.417057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.417059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.417397 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.417568 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.417568 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.417569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.417673 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.418000 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.418006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.418156 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.418604 (+ 448us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.418889 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":163,"thread_start_us":97,"threads_started":1}
I20260928 21:49:24.419013 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.410218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.3.0.252:39847)
0928 21:49:24.410380 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.410388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.410412 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:49:24.410818 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.410824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.410869 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.410992 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.411560 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.412529 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.413224 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.413609 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.413706 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.416155 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.416160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.416160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.416166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.417717 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.418034 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.418036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.418037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.418080 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.418245 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.418246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.418246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.418351 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.418452 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.418578 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.418832 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260928 21:49:24.419446 9886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.252@KRBTEST.COM for oryx/127.3.0.253@KRBTEST.COM
I20260928 21:49:24.425355 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.419704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:38339 (local address 127.0.0.1:47996)
0928 21:49:24.419798 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.419807 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.419852 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.419964 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.419965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.420082 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.420580 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.420582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.421209 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.421210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.421693 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.421695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.422027 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.422028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.422086 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.422369 (+ 283us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.422372 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.423357 (+ 985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.424532 (+ 1175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.424533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.424533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.424676 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.424833 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.424833 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.424834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.424884 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.425086 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.425086 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.425124 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.425184 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.425265 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 21:49:24.425364 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.419761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.3.0.253:38339)
0928 21:49:24.419798 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.419800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.419839 (+ 39us) server_negotiation.cc:407] Connection header received
0928 21:49:24.419909 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.419910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.419920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.419963 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.420654 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.421145 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.421761 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.421973 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.422025 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.423438 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.423440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.423440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.423448 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.424443 (+ 995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.424734 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.424735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.424735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.424783 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.424932 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.424932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.424933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.425036 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.425087 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.425211 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.425281 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:49:24.425733 9886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.252" port: 39847 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.252@KRBTEST.COM for oryx/127.3.0.252@KRBTEST.COM
I20260928 21:49:24.430775 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.425934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:39847 (local address 127.0.0.1:40596)
0928 21:49:24.425969 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.425978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.426011 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.426103 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.426104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.426214 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.426235 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.426236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.426788 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.426789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.427239 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.427240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.427575 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.427575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.427617 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.427873 (+ 256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.427876 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.428910 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.430104 (+ 1194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.430105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.430105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.430245 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.430374 (+ 129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.430375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.430375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.430414 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.430562 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.430562 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.430592 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.430634 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.430699 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 21:49:24.430821 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.425976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40596 (local address 127.3.0.252:39847)
0928 21:49:24.426005 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.426007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.426013 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:49:24.426058 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.426058 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.426065 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.426102 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.426289 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.426729 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.427299 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.427503 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.427575 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.428984 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.428985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.428985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.428991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.430046 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.430296 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.430297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.430297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.430334 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.430447 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.430447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.430447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.430524 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.430563 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.430649 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.430737 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":16}
I20260928 21:49:24.432080 9886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: Bootstrap starting.
I20260928 21:49:24.432889 9886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:24.433130 9886 log.cc:824] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:24.433748 9886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: No bootstrap required, opened a new log
I20260928 21:49:24.435060 9886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 39847 } }
I20260928 21:49:24.435178 9886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:24.435209 9886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e4afeb2e034cc1be7c21e90c025570, State: Initialized, Role: FOLLOWER
I20260928 21:49:24.435316 9886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 39847 } }
I20260928 21:49:24.435536 9891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 39847 } } }
I20260928 21:49:24.435616 9886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:24.435629 9891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:49:24.437018 9902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:49:24.437069 9902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:49:24.439095 9818 master_runner.cc:428] Detected that this master 73e4afeb2e034cc1be7c21e90c025570 is joining an existing cluster
I20260928 21:49:24.439141 9818 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.0.252:39847
I20260928 21:49:24.467209 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.440332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40606 (local address 127.3.0.252:39847)
0928 21:49:24.440362 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.440364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.440681 (+ 317us) server_negotiation.cc:407] Connection header received
0928 21:49:24.440749 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.440750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.440756 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.440778 (+ 22us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.441140 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.441843 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.442971 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.443212 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.443263 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.444162 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.444163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.444163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.444169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.466136 (+ 21967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.466482 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.466483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.466484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.466531 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.466691 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.466692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.466692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.466812 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.466848 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.467031 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.467108 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:49:24.467555 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.440311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:39847 (local address 127.0.0.1:40606)
0928 21:49:24.440638 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.440645 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.440698 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.440821 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.440822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.441027 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.441075 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.441076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.441947 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.441948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.442906 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.442908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.443314 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.443315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.443366 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.443790 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.443793 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.444085 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.466225 (+ 22140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.466226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.466226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.466416 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.466590 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.466590 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.466590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.466637 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.466882 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.466883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.466916 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.467326 (+ 410us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.467459 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":64,"thread_start_us":53,"threads_started":1}
I20260928 21:49:24.467855 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.440441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.3.0.253:38339)
0928 21:49:24.440501 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.440503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.440511 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:24.440532 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.440533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.440542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.440589 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.441149 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.441865 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.443196 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.443485 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.443817 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.465150 (+ 21333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.465152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.465153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.465162 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.466647 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.467009 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.467010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.467010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.467065 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.467265 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.467266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.467266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.467395 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.467458 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.467689 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.467766 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:49:24.467854 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.440074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:38339 (local address 127.0.0.1:48006)
0928 21:49:24.440236 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.440248 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.440315 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.440634 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.440636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.440991 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.441067 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.441068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.441950 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.441951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.443125 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.443128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.443896 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.443897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.443961 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.444335 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.444338 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.465011 (+ 20673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.466732 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.466733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.466734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.466923 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.467135 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.467135 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.467135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.467194 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.467505 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.467506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.467560 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.467648 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.467744 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":66,"spinlock_wait_cycles":9600,"thread_start_us":108,"threads_started":1}
I20260928 21:49:24.469411 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.440074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:59556)
0928 21:49:24.440505 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.440514 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.440569 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.440740 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.440740 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.441005 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.441070 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.441072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.442002 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.442003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.442934 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.442937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.443481 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.443482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.443538 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.465159 (+ 21621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.465163 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.466193 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.468414 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.468415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.468415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.468626 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.468840 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.468840 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.468841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.468892 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.469120 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.469121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.469160 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.469212 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.469307 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":51,"thread_start_us":87,"threads_started":1}
I20260928 21:49:24.469412 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.440448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.3.0.254:46579)
0928 21:49:24.440505 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.440507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.440561 (+ 54us) server_negotiation.cc:407] Connection header received
0928 21:49:24.440628 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.440629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.440638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.440682 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.441163 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.441898 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.443098 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.443376 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.443432 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.466312 (+ 22880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.466314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.466314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.466323 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.467820 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.468714 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.468716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.468716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.468773 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.468958 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.468959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.468959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.469063 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.469116 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.469255 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.469325 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:49:24.470242 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 9228
I20260928 21:49:24.474826 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:38505
--local_ip_for_outbound_sockets=127.3.0.193
--tserver_master_addrs=127.3.0.254:46579
--webserver_port=41685
--webserver_interface=127.3.0.193
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab
--principal=oryx/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.0.254:46579,127.3.0.253:38339
--tserver_master_addrs=127.3.0.254:46579,127.3.0.253:38339,127.3.0.252:39847 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632163, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.254@KRBTEST.COM for oryx/127.3.0.252@KRBTEST.COM
I20260928 21:49:24.478873 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.471637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.3.0.252:39847)
0928 21:49:24.471668 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.471670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.471676 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:49:24.471704 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.471705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.471713 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.471752 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.472084 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.472830 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.473841 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.474154 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.474207 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.476277 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.476278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.476279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.476285 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.477710 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.478102 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.478103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.478103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.478157 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.478367 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.478368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.478368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.478493 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.478560 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.478703 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.478779 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:49:24.478885 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.471448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:39847 (local address 127.0.0.1:40608)
0928 21:49:24.471522 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.471531 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.471565 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.471795 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.471796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.471950 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.472010 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.472012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.472917 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.472919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.473752 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.473754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.474254 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.474255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.474319 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.474762 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.474767 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.476177 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.477799 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.477800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.477800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.478014 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.478224 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.478226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.478226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.478296 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.478556 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.478557 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.478604 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.478672 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.478784 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 21:49:24.479674 9166 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.0.252:39847: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570"
member_type: NON_VOTER
last_known_addr {
host: "127.3.0.252"
port: 39847
}
attrs {
promote: true
}
}
dest_uuid: "06fcd49e1d8745739eb7a281ab96ea8e"
cas_config_opid_index: 5
I20260928 21:49:24.479924 9166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:24.480396 9726 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 73e4afeb2e034cc1be7c21e90c025570 (127.3.0.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.481371 9726 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.0.252:39847
I20260928 21:49:24.481443 9726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.481508 9726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:24.481849 9860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:24.483160 9860 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [term 1 FOLLOWER]: Refusing update from remote peer 06fcd49e1d8745739eb7a281ab96ea8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 21:49:24.483448 9725 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:24.483770 9818 master.cc:537] Master@127.3.0.252:39847 shutting down...
I20260928 21:49:24.484368 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.480356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:38339 (local address 127.0.0.1:48008)
0928 21:49:24.480746 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.480757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.480806 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.481032 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.481033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.481034 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:24.481076 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.481078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.481973 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.481974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.483206 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.483209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.484112 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.484113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.484180 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.484202 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.484253 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.484299 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375}
I20260928 21:49:24.484589 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.480424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.3.0.253:38339)
0928 21:49:24.480715 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.480717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.480854 (+ 137us) server_negotiation.cc:407] Connection header received
0928 21:49:24.480880 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.480881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.480904 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.480971 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:24.481145 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.481863 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.483304 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.483889 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.483984 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.484075 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.484484 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.484526 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260928 21:49:24.485185 9774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 LEARNER]: Refusing update from remote peer 06fcd49e1d8745739eb7a281ab96ea8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260928 21:49:24.485276 9152 consensus_peers.cc:487] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e -> Peer 73e4afeb2e034cc1be7c21e90c025570 (127.3.0.252:39847): Couldn't send request to peer 73e4afeb2e034cc1be7c21e90c025570. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:49:24.485520 9725 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 21:49:24.486088 9774 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 73e4afeb2e034cc1be7c21e90c025570 (127.3.0.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.487112 9803 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06fcd49e1d8745739eb7a281ab96ea8e. Latest consensus state: current_term: 1 leader_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.487185 9803 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:49:24.487213 9803 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: NON_VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: true } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.487248 9803 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:49:24.487982 9725 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e: attempting to promote NON_VOTER f2baa60ebbbf48aa8aa6636e8a5f6587 to VOTER
I20260928 21:49:24.488106 9818 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:24.488191 9818 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:24.488235 9818 tablet_replica.cc:330] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: stopping tablet replica
I20260928 21:49:24.488202 9725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:24.488955 9774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 LEARNER]: Refusing update from remote peer 06fcd49e1d8745739eb7a281ab96ea8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260928 21:49:24.489320 9152 consensus_peers.cc:487] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e -> Peer 73e4afeb2e034cc1be7c21e90c025570 (127.3.0.252:39847): Couldn't send request to peer 73e4afeb2e034cc1be7c21e90c025570. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:49:24.489390 9914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 21:49:24.489902 9726 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f2baa60ebbbf48aa8aa6636e8a5f6587 (127.3.0.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.490765 9726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.490839 9726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:24.490881 9726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.490917 9726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:24.491168 9774 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f2baa60ebbbf48aa8aa6636e8a5f6587 (127.3.0.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.492797 9913 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.493039 9913 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:49:24.503684 9818 master.cc:552] Master@127.3.0.252:39847 shutdown complete.
I20260928 21:49:24.503738 9818 master_runner.cc:305] Clearing existing system tablet
I20260928 21:49:24.503861 9818 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:24.503909 9818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:24.504655 9818 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260928 21:49:24.505122 9917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.505352 9818 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:24.505422 9818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:49:24.505450 9818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal
uuid: "73e4afeb2e034cc1be7c21e90c025570"
format_stamp: "Formatted at 2026-09-28 21:49:24 on dist-test-slave-jrck"
server_key: "6b7aeae5da933a1af68b2e34c9c95d40"
server_key_iv: "24f0e946df2c6698c3f038793e37c10e"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.506174 9818 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:24.507483 9818 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:49:24.507545 9818 log.cc:1196] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 21:49:24.507721 9818 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: Deleting consensus metadata
I20260928 21:49:24.507975 9818 master_runner.cc:315] Copying system tablet from 127.3.0.254:46579
I20260928 21:49:24.508555 9818 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: tablet copy: Beginning tablet copy session from remote peer at address 127.3.0.254:46579
I20260928 21:49:24.519761 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.512688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:59560)
0928 21:49:24.512847 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.512858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.512938 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.513103 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.513103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.513292 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.513355 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.513357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.514292 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.514293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.515136 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.515139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.515624 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.515626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.515690 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.516278 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.516283 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.516736 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.518674 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.518675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.518676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.518879 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.519119 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.519119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.519120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.519182 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.519448 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.519449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.519496 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.519563 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.519661 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260928 21:49:24.519775 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.512747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.3.0.254:46579)
0928 21:49:24.512795 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.512797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.512939 (+ 142us) server_negotiation.cc:407] Connection header received
0928 21:49:24.513003 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.513004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.513017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.513082 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.513439 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.514203 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.515227 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.515539 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.515614 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.516826 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.516828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.516828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.516835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.518542 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.518977 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.518979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.518979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.519054 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.519245 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.519246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.519246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.519381 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.519455 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.519608 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.519688 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:49:24.520515 9186 tablet_copy_service.cc:140] P 06fcd49e1d8745739eb7a281ab96ea8e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 73e4afeb2e034cc1be7c21e90c025570 ({username='oryx', principal='oryx/127.3.0.252@KRBTEST.COM'} at 127.0.0.1:59560)
I20260928 21:49:24.520570 9186 tablet_copy_service.cc:161] P 06fcd49e1d8745739eb7a281ab96ea8e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 73e4afeb2e034cc1be7c21e90c025570 at {username='oryx', principal='oryx/127.3.0.252@KRBTEST.COM'} at 127.0.0.1:59560: session id = 73e4afeb2e034cc1be7c21e90c025570-00000000000000000000000000000000
I20260928 21:49:24.521117 9186 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:49:24.521564 9818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:49:24.522873 9818 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: tablet copy: Starting download of 0 data blocks...
I20260928 21:49:24.523037 9818 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: tablet copy: Starting download of 1 WAL segments...
I20260928 21:49:24.523741 9818 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:49:24.524684 9186 tablet_copy_service.cc:342] P 06fcd49e1d8745739eb7a281ab96ea8e: Request end of tablet copy session 73e4afeb2e034cc1be7c21e90c025570-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.0.252@KRBTEST.COM'} at 127.0.0.1:59560
I20260928 21:49:24.524735 9186 tablet_copy_service.cc:433] P 06fcd49e1d8745739eb7a281ab96ea8e: ending tablet copy session 73e4afeb2e034cc1be7c21e90c025570-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 73e4afeb2e034cc1be7c21e90c025570
I20260928 21:49:24.527480 9818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:24.528535 9924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.528582 9925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.528972 9927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:24.529258 9818 server_base.cc:1034] running on GCE node
I20260928 21:49:24.529415 9818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:24.529536 9818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:24.530638 9818 hybrid_clock.cc:648] HybridClock initialized: now 1790632164530631 us; error 27 us; skew 500 ppm
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:24.532085 9818 init.cc:377] Logged in from keytab as oryx/127.3.0.252@KRBTEST.COM (short username oryx)
I20260928 21:49:24.532325 9818 webserver.cc:460] Webserver started at http://127.3.0.252:38393/ using document root <none> and password file <none>
I20260928 21:49:24.532418 9818 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:24.532440 9818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:24.533264 9818 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:24.533771 9934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.533892 9818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.533944 9818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal
uuid: "73e4afeb2e034cc1be7c21e90c025570"
format_stamp: "Formatted at 2026-09-28 21:49:24 on dist-test-slave-jrck"
server_key: "6b7aeae5da933a1af68b2e34c9c95d40"
server_key_iv: "24f0e946df2c6698c3f038793e37c10e"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.533990 9818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:24.541860 9818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:24.543538 9818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:24.546440 9818 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.252:39847
I20260928 21:49:24.546531 9986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.252:39847 every 8 connection(s)
I20260928 21:49:24.546864 9818 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
I20260928 21:49:24.547755 9987 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:49:24.548242 9987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: Bootstrap starting.
I20260928 21:49:24.551127 9987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:49:24.551321 9987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: Bootstrap complete.
I20260928 21:49:24.551502 9987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:24.551592 9987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73e4afeb2e034cc1be7c21e90c025570, State: Initialized, Role: LEARNER
I20260928 21:49:24.551626 9987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:24.551788 9987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:24.551791 9990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } } }
I20260928 21:49:24.552011 9990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:49:24.553543 10001 catalog_manager.cc:1259] Loaded cluster ID: 1fac966ae3204ced96d738c1f76e5f5b
I20260928 21:49:24.553627 10001 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: loading cluster ID for follower catalog manager: success
I20260928 21:49:24.554821 10001 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: acquiring CA information for follower catalog manager: success
I20260928 21:49:24.555135 10001 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73e4afeb2e034cc1be7c21e90c025570: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 21:49:24.573666 9911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:24.573858 9911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:24.573879 9911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:24.574985 9911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:24.575023 9911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:24.575052 9911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:24.576468 9911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab
--principal=oryx/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:38505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=41685
--tserver_master_addrs=127.3.0.254:46579,127.3.0.253:38339,127.3.0.252:39847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:24.576683 9911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:24.576889 9911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:24.581583 10009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.581525 10007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.581525 10006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:24.581552 9911 server_base.cc:1034] running on GCE node
I20260928 21:49:24.581925 9911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:24.582147 9911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:24.583278 9911 hybrid_clock.cc:648] HybridClock initialized: now 1790632164583263 us; error 33 us; skew 500 ppm
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:24.585590 9911 init.cc:377] Logged in from keytab as oryx/127.3.0.193@KRBTEST.COM (short username oryx)
I20260928 21:49:24.585981 9911 webserver.cc:460] Webserver started at http://127.3.0.193:41685/ using document root <none> and password file <none>
I20260928 21:49:24.586180 9911 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:24.586239 9911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:24.587888 9911 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:24.588394 10016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.588552 9911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.588630 9911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "645f2026f84d4693afbf688a5961bd79"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "daad60fb030093ceb365a258f3e17acc"
server_key_iv: "21b516939694920314e2116e9ddcc0cb"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.588949 9911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:24.607323 9911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:24.608979 9911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:24.609113 9911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:24.609306 9911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:24.609577 9911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:24.609608 9911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.609635 9911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:24.609658 9911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.615437 9911 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:38505
I20260928 21:49:24.615521 10129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:38505 every 8 connection(s)
I20260928 21:49:24.615811 9911 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:49:24.619931 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 9911
I20260928 21:49:24.620085 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 9362
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.253@KRBTEST.COM
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.252@KRBTEST.COM
I20260928 21:49:24.626864 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.616787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:55725 (local address 127.3.0.254:46579)
0928 21:49:24.616839 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.616841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.616996 (+ 155us) server_negotiation.cc:407] Connection header received
0928 21:49:24.618074 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.618075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.618087 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.618134 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.618808 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.619317 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.620732 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.621038 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.621079 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.624037 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.624039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.624039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.624048 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.625410 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.625853 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.625854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.625855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.625911 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.626183 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.626184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.626184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.626310 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.626348 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.626702 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.626779 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:24.626865 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.616804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:60811 (local address 127.3.0.253:38339)
0928 21:49:24.616851 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.616853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.616988 (+ 135us) server_negotiation.cc:407] Connection header received
0928 21:49:24.618001 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.618002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.618013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.618066 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.618833 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.619352 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.620746 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.621014 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.621054 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.623807 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.623809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.623809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.623817 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.625290 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.625836 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.625838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.625838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.625895 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.626198 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.626199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.626199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.626341 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.626382 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.626704 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.626775 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:24.627272 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:35407
--local_ip_for_outbound_sockets=127.3.0.194
--tserver_master_addrs=127.3.0.254:46579
--webserver_port=43779
--webserver_interface=127.3.0.194
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab
--principal=oryx/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.0.254:46579,127.3.0.253:38339
--tserver_master_addrs=127.3.0.254:46579,127.3.0.253:38339,127.3.0.252:39847 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:49:24.627349 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.616560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:55725)
0928 21:49:24.616916 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.616925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.617913 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.618157 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.618158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.618522 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.618725 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.618731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.619414 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.619415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.620650 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.620651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.621152 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.621153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.621263 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.621767 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.621776 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.623937 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.625500 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.625501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.625502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.625776 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.625972 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.625973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.625973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.626118 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.626388 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.626394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.626559 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.626940 (+ 381us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.627112 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":30,"thread_start_us":90,"threads_started":1}
I20260928 21:49:24.627825 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.616560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:39847 (local address 127.3.0.193:55899)
0928 21:49:24.617091 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.617098 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.617917 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.618262 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.618263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.618558 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.618725 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.618730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.620618 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.620619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.621370 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.621372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.621777 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.621778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.621838 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.622499 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.622504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.625013 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.626755 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.626756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.626756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.626995 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.627190 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.627191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.627191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.627257 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.627524 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.627525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.627562 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.627625 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.627725 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"spinlock_wait_cycles":18688,"thread_start_us":84,"threads_started":1}
I20260928 21:49:24.627848 10137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.616621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:55899 (local address 127.3.0.252:39847)
0928 21:49:24.617202 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.617204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.617212 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:24.618186 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.618188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.618200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.618261 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.619408 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.620530 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.621453 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.621710 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.621773 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.625108 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.625109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.625109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.625116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.626685 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.627076 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.627077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.627077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.627134 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.627310 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.627311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.627311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.627435 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.627502 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.627679 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.627764 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":58,"threads_started":1}
I20260928 21:49:24.628183 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.616560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:38339 (local address 127.3.0.193:60811)
0928 21:49:24.616871 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.616910 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.617910 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.618132 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.618137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.618584 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.618725 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.618730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.619430 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.619431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.620651 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.620654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.621134 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.621135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.621263 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.621986 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.621990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.623670 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.625370 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.625376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.625379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.625749 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.625966 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.625966 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.625968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.626119 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.626425 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.626426 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.626557 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.627672 (+ 1115us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.627807 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":60,"thread_start_us":83,"threads_started":1}
I20260928 21:49:24.628592 10132 heartbeater.cc:347] Connected to a master server at 127.3.0.253:38339
I20260928 21:49:24.628682 10132 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:24.628865 10132 heartbeater.cc:511] Master 127.3.0.253:38339 requested a full tablet report, sending...
I20260928 21:49:24.628916 10130 heartbeater.cc:347] Connected to a master server at 127.3.0.252:39847
I20260928 21:49:24.628980 10130 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:24.629122 10130 heartbeater.cc:511] Master 127.3.0.252:39847 requested a full tablet report, sending...
I20260928 21:49:24.629354 9764 ts_manager.cc:194] Registered new tserver with Master: 645f2026f84d4693afbf688a5961bd79 (127.3.0.193:38505)
I20260928 21:49:24.629441 9945 ts_manager.cc:194] Registered new tserver with Master: 645f2026f84d4693afbf688a5961bd79 (127.3.0.193:38505)
I20260928 21:49:24.628954 10131 heartbeater.cc:347] Connected to a master server at 127.3.0.254:46579
I20260928 21:49:24.629501 10131 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:24.629565 10131 heartbeater.cc:511] Master 127.3.0.254:46579 requested a full tablet report, sending...
I20260928 21:49:24.629812 9166 ts_manager.cc:194] Re-registered known tserver with Master: 645f2026f84d4693afbf688a5961bd79 (127.3.0.193:38505)
I20260928 21:49:24.630164 9166 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:55725
W20260928 21:49:24.706658 10138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:24.706858 10138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:24.706888 10138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:24.708007 10138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:24.708045 10138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:24.708078 10138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:24.709544 10138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab
--principal=oryx/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:35407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=43779
--tserver_master_addrs=127.3.0.254:46579,127.3.0.253:38339,127.3.0.252:39847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:24.709779 10138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:24.709972 10138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:24.714582 10146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.714571 10143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.715242 10144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:24.715996 10138 server_base.cc:1034] running on GCE node
I20260928 21:49:24.716166 10138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:24.716374 10138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:24.717512 10138 hybrid_clock.cc:648] HybridClock initialized: now 1790632164717499 us; error 30 us; skew 500 ppm
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:24.719870 10138 init.cc:377] Logged in from keytab as oryx/127.3.0.194@KRBTEST.COM (short username oryx)
I20260928 21:49:24.720321 10138 webserver.cc:460] Webserver started at http://127.3.0.194:43779/ using document root <none> and password file <none>
I20260928 21:49:24.720557 10138 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:24.720623 10138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:24.721864 10138 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:24.722388 10153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.722558 10138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.722640 10138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "6f2ec22333cb388e93d3291a83cd2d08"
server_key_iv: "a7ede8c5063673ce830307bd0e7091af"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.722965 10138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:24.741616 10138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:24.743822 10138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:24.743964 10138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:24.744180 10138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:24.744503 10138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:24.744541 10138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.744571 10138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:24.744594 10138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.749954 10138 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:35407
I20260928 21:49:24.750020 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:35407 every 8 connection(s)
I20260928 21:49:24.750324 10138 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:49:24.752310 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 10138
I20260928 21:49:24.752501 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 9496
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.252@KRBTEST.COM
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:24.758234 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:42211
--local_ip_for_outbound_sockets=127.3.0.195
--tserver_master_addrs=127.3.0.254:46579
--webserver_port=43805
--webserver_interface=127.3.0.195
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab
--principal=oryx/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.0.254:46579,127.3.0.253:38339
--tserver_master_addrs=127.3.0.254:46579,127.3.0.253:38339,127.3.0.252:39847 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.253@KRBTEST.COM
I20260928 21:49:24.761305 10137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.751088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:47343 (local address 127.3.0.252:39847)
0928 21:49:24.751283 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.751285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.751463 (+ 178us) server_negotiation.cc:407] Connection header received
0928 21:49:24.751976 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.751978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.752002 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.752030 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.753116 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.753819 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.755167 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.755484 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.755545 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.758198 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.758200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.758200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.758208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.759694 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.760156 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.760158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.760158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.760214 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.760561 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.760562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.760563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.760695 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.760748 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.761125 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.761215 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260928 21:49:24.761564 10272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.750985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:47435)
0928 21:49:24.751335 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.751348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.751866 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.752128 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.752133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.752592 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.752870 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.752874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.753823 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.753824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.755051 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.755053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.755572 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.755572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.755707 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.756290 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.756295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.758568 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.760345 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.760346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.760346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.760592 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.760772 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.760773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.760773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.760826 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.761086 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.761087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.761121 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.761177 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.761331 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260928 21:49:24.761762 10271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.750986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:39847 (local address 127.3.0.194:47343)
0928 21:49:24.751316 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.751347 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.751861 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.752128 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.752133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.752960 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.753027 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.753029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.754042 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.754043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.755096 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.755098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.755602 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.755603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.755707 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.756265 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.756274 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.758093 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.759781 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.759787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.759789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.760068 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.760281 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.760282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.760282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.760479 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.760792 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.760798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.760966 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.761470 (+ 504us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.761552 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260928 21:49:24.761992 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.751147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:47435 (local address 127.3.0.254:46579)
0928 21:49:24.751245 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.751247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.751440 (+ 193us) server_negotiation.cc:407] Connection header received
0928 21:49:24.751978 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.751979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.751992 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.752024 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.752976 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.753708 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.755151 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.755430 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.755471 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.758663 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.758664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.758664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.758674 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.760254 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.760659 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.760660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.760661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.760712 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.760885 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.760886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.760886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.761006 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.761063 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.761827 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.761907 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 21:49:24.762468 10267 heartbeater.cc:347] Connected to a master server at 127.3.0.252:39847
I20260928 21:49:24.762554 10267 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:24.762689 10267 heartbeater.cc:511] Master 127.3.0.252:39847 requested a full tablet report, sending...
I20260928 21:49:24.762859 10268 heartbeater.cc:347] Connected to a master server at 127.3.0.254:46579
I20260928 21:49:24.762907 10268 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:24.762971 10268 heartbeater.cc:511] Master 127.3.0.254:46579 requested a full tablet report, sending...
I20260928 21:49:24.763089 9945 ts_manager.cc:194] Registered new tserver with Master: 7f84f2e6c1ea499aabc1b2c6cf406f11 (127.3.0.194:35407)
I20260928 21:49:24.763265 9166 ts_manager.cc:194] Re-registered known tserver with Master: 7f84f2e6c1ea499aabc1b2c6cf406f11 (127.3.0.194:35407)
I20260928 21:49:24.763664 9166 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:47435
I20260928 21:49:24.764048 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.751037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:54567 (local address 127.3.0.253:38339)
0928 21:49:24.751193 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.751196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.751588 (+ 392us) server_negotiation.cc:407] Connection header received
0928 21:49:24.751949 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.751950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.751962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.752011 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.755784 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.756923 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.757596 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.757856 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.757916 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.760858 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.760859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.760860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.760866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.762754 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.763233 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.763235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.763235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.763292 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.763494 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.763495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.763496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.763639 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.763712 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.763884 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.763959 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260928 21:49:24.764614 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.750986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:38339 (local address 127.3.0.194:54567)
0928 21:49:24.751499 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.751507 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.751869 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.752567 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.752568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.752765 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.752870 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.752875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.757000 (+ 4125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.757001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.757522 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.757524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.758032 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.758033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.758097 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.758814 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.758819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.760761 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.762840 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.762841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.762841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.763087 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.763364 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.763365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.763365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.763420 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.763712 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.763712 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.763762 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.764153 (+ 391us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.764279 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":28,"spinlock_wait_cycles":40320,"thread_start_us":105,"threads_started":1}
I20260928 21:49:24.764952 10269 heartbeater.cc:347] Connected to a master server at 127.3.0.253:38339
I20260928 21:49:24.764994 10269 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:24.765056 10269 heartbeater.cc:511] Master 127.3.0.253:38339 requested a full tablet report, sending...
I20260928 21:49:24.765332 9764 ts_manager.cc:194] Registered new tserver with Master: 7f84f2e6c1ea499aabc1b2c6cf406f11 (127.3.0.194:35407)
W20260928 21:49:24.837873 10274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:24.838064 10274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:24.838085 10274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:24.839147 10274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:24.839180 10274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:24.839205 10274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:24.840596 10274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab
--principal=oryx/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:42211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=43805
--tserver_master_addrs=127.3.0.254:46579,127.3.0.253:38339,127.3.0.252:39847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:24.840766 10274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:24.840931 10274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:24.845435 10279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:24.845441 10282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:24.845544 10274 server_base.cc:1034] running on GCE node
W20260928 21:49:24.845463 10280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:24.845811 10274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:24.846011 10274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:24.847142 10274 hybrid_clock.cc:648] HybridClock initialized: now 1790632164847128 us; error 28 us; skew 500 ppm
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:24.849380 10274 init.cc:377] Logged in from keytab as oryx/127.3.0.195@KRBTEST.COM (short username oryx)
I20260928 21:49:24.849751 10274 webserver.cc:460] Webserver started at http://127.3.0.195:43805/ using document root <none> and password file <none>
I20260928 21:49:24.849931 10274 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:24.850000 10274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:24.851202 10274 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:24.851677 10289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.851802 10274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.851894 10274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "d046de81da1246bfb25cf3bbe0a92eb9"
format_stamp: "Formatted at 2026-09-28 21:49:23 on dist-test-slave-jrck"
server_key: "08d86712f45cab0ed9971cfedca4397a"
server_key_iv: "748823baf69addb8f394845b9d63428b"
server_key_version: "encryptionkey@0"
I20260928 21:49:24.852226 10274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:24.864403 10274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:24.865984 10274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:24.866092 10274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:24.866256 10274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:49:24.866482 10274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:24.866508 10274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.866528 10274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:24.866544 10274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:24.872242 10274 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:42211
I20260928 21:49:24.872295 10402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:42211 every 8 connection(s)
I20260928 21:49:24.872630 10274 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:49:24.873143 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 10274
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.253@KRBTEST.COM
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
Sep 28 21:49:24 dist-test-slave-jrck krb5kdc[9121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632164, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.252@KRBTEST.COM
I20260928 21:49:24.882189 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.874267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.3.0.252:39847)
0928 21:49:24.874366 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.874367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.874605 (+ 238us) server_negotiation.cc:407] Connection header received
0928 21:49:24.874630 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.874631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.874642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.874691 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.875486 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.876242 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.877253 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.877551 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.877601 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.879044 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.879046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.879046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.879053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.880798 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.881165 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.881166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.881166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.881229 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.881418 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.881419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.881420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.881561 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.881653 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.882013 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.882090 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":42,"threads_started":1}
I20260928 21:49:24.885140 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.874328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59568 (local address 127.3.0.254:46579)
0928 21:49:24.874452 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.874454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.874555 (+ 101us) server_negotiation.cc:407] Connection header received
0928 21:49:24.874583 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.874584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.874596 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.874652 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.875900 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.876639 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.877586 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.878136 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.878187 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.880577 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.880578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.880578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.880585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.883801 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.884149 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.884150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.884150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.884203 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.884372 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.884372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.884373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.884527 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.884613 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.884960 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.885043 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260928 21:49:24.885259 10137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.874048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:48201 (local address 127.3.0.252:39847)
0928 21:49:24.874088 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.874090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.874234 (+ 144us) server_negotiation.cc:407] Connection header received
0928 21:49:24.874967 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.874968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.874977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.875016 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.875750 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.876512 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.879434 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.879709 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.879751 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.882432 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.882433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.882433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.882439 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.883790 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.884244 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.884245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.884245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.884297 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.884595 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.884596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.884596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.884711 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.884749 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.885105 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.885179 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:49:24.885768 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.873974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:39847 (local address 127.3.0.195:48201)
0928 21:49:24.874139 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.874148 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.874902 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.875053 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.875054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.875383 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.875648 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.875653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.876620 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.876620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.879346 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.879348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.879821 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.879822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.879892 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.880575 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.880580 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.882009 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.883877 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.883885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.883887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.884158 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.884363 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.884364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.884364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.884524 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.884791 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.884799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.884962 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.885369 (+ 407us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.885552 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1818,"thread_start_us":68,"threads_started":1}
I20260928 21:49:24.886157 10403 heartbeater.cc:347] Connected to a master server at 127.3.0.252:39847
I20260928 21:49:24.886246 10403 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:24.886420 10403 heartbeater.cc:511] Master 127.3.0.252:39847 requested a full tablet report, sending...
I20260928 21:49:24.886433 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.873790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:36809 (local address 127.3.0.253:38339)
0928 21:49:24.873832 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.873834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.873843 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:49:24.874812 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.874813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.874823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.874875 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.875745 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.876511 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.877937 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.878234 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.878289 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.882185 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.882186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.882186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.882192 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.883790 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.884805 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.884806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.884806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.884855 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.885146 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.885147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.885147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.885261 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.885313 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.886284 (+ 971us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.886354 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:24.886806 9945 ts_manager.cc:194] Registered new tserver with Master: d046de81da1246bfb25cf3bbe0a92eb9 (127.3.0.195:42211)
I20260928 21:49:24.888026 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.873276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:50495)
0928 21:49:24.873441 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.873481 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.874671 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.874967 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.874971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.876901 (+ 1930us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.876961 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.876963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.877870 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.877871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.878602 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.878604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.879083 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.879084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.879142 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.880038 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.880042 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.882393 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.885575 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.885576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.885576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.885759 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.887395 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.887395 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.887396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.887445 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.887731 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.887732 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.887774 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.887839 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.887924 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260928 21:49:24.888038 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.873507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:50495 (local address 127.3.0.254:46579)
0928 21:49:24.873554 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.873556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.873566 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:49:24.874792 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.874793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.874805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.874833 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.877055 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.877782 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.878691 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.878978 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.879015 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.882485 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.882487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.882487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.882506 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.884485 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.887267 (+ 2782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.887268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.887269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.887324 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.887516 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.887516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.887517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.887643 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.887685 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.887886 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.887963 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:49:24.888622 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.873276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:38339 (local address 127.3.0.195:36809)
0928 21:49:24.873532 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.873542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.874677 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.874967 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.874971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.875362 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:24.875648 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.875653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.876620 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.876620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.877861 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.877863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.878342 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.878343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.878469 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.879346 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:24.879354 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:24.882114 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:24.884558 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.884558 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.884559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.884735 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.885029 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:24.885029 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:24.885030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:24.885085 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:24.885945 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:24.885946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:24.885989 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.886103 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.888538 (+ 2435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":29,"thread_start_us":80,"threads_started":1}
I20260928 21:49:24.886863 9166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:24.888893 9166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:24.889092 10404 heartbeater.cc:347] Connected to a master server at 127.3.0.254:46579
I20260928 21:49:24.889137 10404 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:24.889158 10405 heartbeater.cc:347] Connected to a master server at 127.3.0.253:38339
I20260928 21:49:24.889185 10405 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:24.889212 10404 heartbeater.cc:511] Master 127.3.0.254:46579 requested a full tablet report, sending...
I20260928 21:49:24.889225 10405 heartbeater.cc:511] Master 127.3.0.253:38339 requested a full tablet report, sending...
I20260928 21:49:24.889411 9165 ts_manager.cc:194] Re-registered known tserver with Master: d046de81da1246bfb25cf3bbe0a92eb9 (127.3.0.195:42211)
I20260928 21:49:24.889451 9764 ts_manager.cc:194] Registered new tserver with Master: d046de81da1246bfb25cf3bbe0a92eb9 (127.3.0.195:42211)
I20260928 21:49:24.889626 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.874619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.3.0.253:38339)
0928 21:49:24.877900 (+ 3281us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.877902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.878024 (+ 122us) server_negotiation.cc:407] Connection header received
0928 21:49:24.879285 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.879286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.879300 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.879339 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.882730 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.883457 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.884371 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.884670 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.884708 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.886613 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.886615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.886615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.886622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.888143 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.888543 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.888544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.888544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.888599 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.888776 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.888777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.888777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.888887 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.888966 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.889340 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.889415 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3262,"thread_start_us":62,"threads_started":1}
I20260928 21:49:24.889822 9165 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:50495
I20260928 21:49:24.896097 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.892431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:35407 (local address 127.0.0.1:34082)
0928 21:49:24.892660 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.892670 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.892711 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.893066 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.893067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.893068 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:24.893111 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.893113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.894001 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.894002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.894983 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.894985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.895872 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.895873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.895928 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.895932 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.895990 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.896031 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260928 21:49:24.896131 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.892571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.3.0.194:35407)
0928 21:49:24.892734 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.892776 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.892788 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:49:24.892829 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.892831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.892848 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.892922 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:24.893167 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.893934 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.895060 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.895795 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.895867 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.895935 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.896011 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.896053 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260928 21:49:24.896627 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.892881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.3.0.193:38505)
0928 21:49:24.893060 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.893064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.893074 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:49:24.893111 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.893113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.893129 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.893213 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:24.893278 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.894338 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.895584 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.896288 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.896353 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.896420 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.896523 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.896557 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260928 21:49:24.896624 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.892543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:38505 (local address 127.0.0.1:37850)
0928 21:49:24.892668 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.892676 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.892711 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.893211 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.893211 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.893213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:24.893240 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.893241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.894435 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.894436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.895511 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.895514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.896355 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.896356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.896416 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.896420 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.896507 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.896545 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260928 21:49:24.896971 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.892563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:42211 (local address 127.0.0.1:60626)
0928 21:49:24.893007 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.893017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.893058 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.893293 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.893294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.893296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:24.893343 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.893345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.894542 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.894543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.895438 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.895441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.896549 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.896550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.896609 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.896614 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.896641 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.896882 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":92,"threads_started":1}
I20260928 21:49:24.896853 10201 tablet_service.cc:1467] Processing CreateTablet for tablet c43c7076d80c42579da778382062345c (DEFAULT_TABLE table=test-table [id=b5d4cfdc360847ea8891a5288b786e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:24.897205 10201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43c7076d80c42579da778382062345c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:24.897194 10064 tablet_service.cc:1467] Processing CreateTablet for tablet c43c7076d80c42579da778382062345c (DEFAULT_TABLE table=test-table [id=b5d4cfdc360847ea8891a5288b786e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:24.897466 10064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43c7076d80c42579da778382062345c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:24.898000 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.892734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.3.0.195:42211)
0928 21:49:24.893157 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.893161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.893168 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:49:24.893194 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.893196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.893209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.893405 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:24.893443 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.894478 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.895531 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.896105 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.897535 (+ 1430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.897859 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.897886 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.897921 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":71,"threads_started":1}
I20260928 21:49:24.898731 10337 tablet_service.cc:1467] Processing CreateTablet for tablet c43c7076d80c42579da778382062345c (DEFAULT_TABLE table=test-table [id=b5d4cfdc360847ea8891a5288b786e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:24.899039 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43c7076d80c42579da778382062345c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:24.899923 10429 tablet_bootstrap.cc:492] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: Bootstrap starting.
I20260928 21:49:24.900645 10430 tablet_bootstrap.cc:492] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: Bootstrap starting.
I20260928 21:49:24.901051 10429 tablet_bootstrap.cc:654] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:24.901224 10430 tablet_bootstrap.cc:654] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:24.901330 10429 log.cc:824] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:24.901465 10430 log.cc:824] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:24.901853 10433 tablet_bootstrap.cc:492] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: Bootstrap starting.
I20260928 21:49:24.902087 10429 tablet_bootstrap.cc:492] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: No bootstrap required, opened a new log
I20260928 21:49:24.902158 10429 ts_tablet_manager.cc:1397] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:49:24.902400 10433 tablet_bootstrap.cc:654] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:24.902669 10433 log.cc:824] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:24.903383 10433 tablet_bootstrap.cc:492] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: No bootstrap required, opened a new log
I20260928 21:49:24.903448 10433 ts_tablet_manager.cc:1397] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:24.904404 10429 raft_consensus.cc:348] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 } } peers { permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 } } peers { permanent_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 42211 } }
I20260928 21:49:24.904660 10429 raft_consensus.cc:374] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:24.904711 10429 raft_consensus.cc:729] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645f2026f84d4693afbf688a5961bd79, State: Initialized, Role: FOLLOWER
I20260928 21:49:24.904860 10430 tablet_bootstrap.cc:492] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: No bootstrap required, opened a new log
I20260928 21:49:24.904824 10429 consensus_queue.cc:260] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 } } peers { permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 } } peers { permanent_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 42211 } }
I20260928 21:49:24.904918 10430 ts_tablet_manager.cc:1397] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 21:49:24.905058 10429 ts_tablet_manager.cc:1428] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 21:49:24.905200 10131 heartbeater.cc:503] Master 127.3.0.254:46579 was elected leader, sending a full tablet report...
I20260928 21:49:24.905324 10433 raft_consensus.cc:348] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 } } peers { permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 } } peers { permanent_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 42211 } }
I20260928 21:49:24.905462 10433 raft_consensus.cc:374] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:24.905495 10433 raft_consensus.cc:729] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d046de81da1246bfb25cf3bbe0a92eb9, State: Initialized, Role: FOLLOWER
I20260928 21:49:24.905613 10433 consensus_queue.cc:260] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 } } peers { permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 } } peers { permanent_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 42211 } }
I20260928 21:49:24.905825 10433 ts_tablet_manager.cc:1428] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:49:24.906422 10404 heartbeater.cc:503] Master 127.3.0.254:46579 was elected leader, sending a full tablet report...
I20260928 21:49:24.907254 10430 raft_consensus.cc:348] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 } } peers { permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 } } peers { permanent_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 42211 } }
I20260928 21:49:24.907411 10430 raft_consensus.cc:374] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:24.907449 10430 raft_consensus.cc:729] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f84f2e6c1ea499aabc1b2c6cf406f11, State: Initialized, Role: FOLLOWER
I20260928 21:49:24.907554 10430 consensus_queue.cc:260] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 } } peers { permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 } } peers { permanent_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 42211 } }
I20260928 21:49:24.907789 10430 ts_tablet_manager.cc:1428] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:49:24.907840 10268 heartbeater.cc:503] Master 127.3.0.254:46579 was elected leader, sending a full tablet report...
I20260928 21:49:24.921371 10436 raft_consensus.cc:482] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:24.921471 10436 raft_consensus.cc:504] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 } } peers { permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 } } peers { permanent_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 42211 } }
I20260928 21:49:24.921789 10436 leader_election.cc:288] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f84f2e6c1ea499aabc1b2c6cf406f11 (127.3.0.194:35407), 645f2026f84d4693afbf688a5961bd79 (127.3.0.193:38505)
I20260928 21:49:24.924407 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.922094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:44929 (local address 127.3.0.193:38505)
0928 21:49:24.922128 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.922130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.922138 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:24.922163 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.922164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.922173 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.922210 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:24.922343 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.922854 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.923752 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.924117 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.924176 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.924185 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.924315 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.924345 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:49:24.924417 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.922042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:38505 (local address 127.3.0.195:44929)
0928 21:49:24.922071 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.922081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.922117 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.922222 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.922222 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.922223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:24.922290 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.922292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.922932 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.922932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.923674 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.923676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.924185 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.924186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.924257 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.924260 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.924303 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.924341 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":28}
I20260928 21:49:24.924417 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.921987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:57577 (local address 127.3.0.194:35407)
0928 21:49:24.922032 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.922034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.922109 (+ 75us) server_negotiation.cc:407] Connection header received
0928 21:49:24.922167 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.922167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.922179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.922223 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:24.922349 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.922815 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.923755 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.924117 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.924175 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.924185 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.924322 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.924356 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:49:24.924441 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.921949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:35407 (local address 127.3.0.195:57577)
0928 21:49:24.922045 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:24.922057 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:24.922107 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:24.922231 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:24.922232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:24.922233 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:24.922293 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.922294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.922883 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.922884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.923675 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:24.923677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.924183 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.924184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:24.924262 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.924264 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:49:24.924310 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:49:24.924345 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":36}
I20260928 21:49:24.924794 10221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43c7076d80c42579da778382062345c" candidate_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" is_pre_election: true
I20260928 21:49:24.924793 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43c7076d80c42579da778382062345c" candidate_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "645f2026f84d4693afbf688a5961bd79" is_pre_election: true
I20260928 21:49:24.924930 10221 raft_consensus.cc:2454] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d046de81da1246bfb25cf3bbe0a92eb9 in term 0.
I20260928 21:49:24.924938 10084 raft_consensus.cc:2454] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d046de81da1246bfb25cf3bbe0a92eb9 in term 0.
I20260928 21:49:24.925112 10290 leader_election.cc:302] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f84f2e6c1ea499aabc1b2c6cf406f11, d046de81da1246bfb25cf3bbe0a92eb9; no voters:
I20260928 21:49:24.925232 10436 raft_consensus.cc:2781] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:24.925287 10436 raft_consensus.cc:482] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:24.925308 10436 raft_consensus.cc:3037] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:24.926008 10436 raft_consensus.cc:504] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 } } peers { permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 } } peers { permanent_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 42211 } }
I20260928 21:49:24.926141 10436 leader_election.cc:288] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [CANDIDATE]: Term 1 election: Requested vote from peers 7f84f2e6c1ea499aabc1b2c6cf406f11 (127.3.0.194:35407), 645f2026f84d4693afbf688a5961bd79 (127.3.0.193:38505)
I20260928 21:49:24.926256 10221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43c7076d80c42579da778382062345c" candidate_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11"
I20260928 21:49:24.926285 10084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43c7076d80c42579da778382062345c" candidate_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "645f2026f84d4693afbf688a5961bd79"
I20260928 21:49:24.926324 10221 raft_consensus.cc:3037] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:24.926347 10084 raft_consensus.cc:3037] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:24.926976 10221 raft_consensus.cc:2454] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d046de81da1246bfb25cf3bbe0a92eb9 in term 1.
I20260928 21:49:24.927045 10084 raft_consensus.cc:2454] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d046de81da1246bfb25cf3bbe0a92eb9 in term 1.
I20260928 21:49:24.927124 10290 leader_election.cc:302] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f84f2e6c1ea499aabc1b2c6cf406f11, d046de81da1246bfb25cf3bbe0a92eb9; no voters:
I20260928 21:49:24.927217 10436 raft_consensus.cc:2781] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:24.927380 10436 raft_consensus.cc:686] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 1 LEADER]: Becoming Leader. State: Replica: d046de81da1246bfb25cf3bbe0a92eb9, State: Running, Role: LEADER
I20260928 21:49:24.927464 10436 consensus_queue.cc:237] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 } } peers { permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 } } peers { permanent_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 42211 } }
I20260928 21:49:24.928236 9166 catalog_manager.cc:5495] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d046de81da1246bfb25cf3bbe0a92eb9 (127.3.0.195). New cstate: current_term: 1 leader_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d046de81da1246bfb25cf3bbe0a92eb9" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 42211 } health_report { overall_health: HEALTHY } } }
I20260928 21:49:24.932951 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.931011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.3.0.195:42211)
0928 21:49:24.931045 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.931046 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.931053 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:49:24.931094 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.931095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.931109 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.931160 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:24.931242 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.931680 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.932313 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.932515 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.932555 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.932656 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:24.932742 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:24.932819 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.932871 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.932889 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:49:24.934473 10221 raft_consensus.cc:1261] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11 [term 1 FOLLOWER]: Refusing update from remote peer d046de81da1246bfb25cf3bbe0a92eb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:24.934475 10084 raft_consensus.cc:1261] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79 [term 1 FOLLOWER]: Refusing update from remote peer d046de81da1246bfb25cf3bbe0a92eb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:24.934810 10438 consensus_queue.cc:1035] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "645f2026f84d4693afbf688a5961bd79" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:24.934930 10436 consensus_queue.cc:1035] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f84f2e6c1ea499aabc1b2c6cf406f11" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:24.938733 10442 mvcc.cc:205] Tried to move back new op lower bound from 7334429347569430528 to 7334429347543519232. Current Snapshot: MvccSnapshot[applied={T|T < 7334429347569430528}]
I20260928 21:49:24.939919 9165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:24.940001 9165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:24.941179 9165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06fcd49e1d8745739eb7a281ab96ea8e: Sending DeleteTablet for 3 replicas of tablet c43c7076d80c42579da778382062345c
I20260928 21:49:24.941493 10201 tablet_service.cc:1514] Processing DeleteTablet for tablet c43c7076d80c42579da778382062345c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:24 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:34082
I20260928 21:49:24.941542 10064 tablet_service.cc:1514] Processing DeleteTablet for tablet c43c7076d80c42579da778382062345c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:24 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:37850
I20260928 21:49:24.941704 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet c43c7076d80c42579da778382062345c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:24 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:60626
I20260928 21:49:24.941927 10452 tablet_replica.cc:330] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: stopping tablet replica
I20260928 21:49:24.941979 10450 tablet_replica.cc:330] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: stopping tablet replica
I20260928 21:49:24.942072 10450 raft_consensus.cc:2229] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:24.942085 10452 raft_consensus.cc:2229] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:24.942174 10450 raft_consensus.cc:2258] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:24.942284 10452 raft_consensus.cc:2258] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:24.942328 10451 tablet_replica.cc:330] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: stopping tablet replica
I20260928 21:49:24.942409 10451 raft_consensus.cc:2229] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:24.942595 10451 raft_consensus.cc:2258] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:24.942855 10452 ts_tablet_manager.cc:1905] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:24.943009 10451 ts_tablet_manager.cc:1905] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:24.943050 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 9134
I20260928 21:49:24.943794 10450 ts_tablet_manager.cc:1905] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:24.944617 10452 ts_tablet_manager.cc:1918] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:24.944744 10452 log.cc:1196] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/c43c7076d80c42579da778382062345c
I20260928 21:49:24.945041 10452 ts_tablet_manager.cc:1939] T c43c7076d80c42579da778382062345c P d046de81da1246bfb25cf3bbe0a92eb9: Deleting consensus metadata
I20260928 21:49:24.945324 10450 ts_tablet_manager.cc:1918] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:24.945393 10450 log.cc:1196] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/c43c7076d80c42579da778382062345c
I20260928 21:49:24.945629 10450 ts_tablet_manager.cc:1939] T c43c7076d80c42579da778382062345c P 7f84f2e6c1ea499aabc1b2c6cf406f11: Deleting consensus metadata
I20260928 21:49:24.946720 10451 ts_tablet_manager.cc:1918] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:24.946781 10451 log.cc:1196] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/c43c7076d80c42579da778382062345c
I20260928 21:49:24.946990 10451 ts_tablet_manager.cc:1939] T c43c7076d80c42579da778382062345c P 645f2026f84d4693afbf688a5961bd79: Deleting consensus metadata
W20260928 21:49:24.949918 10290 connection.cc:535] server connection from 127.0.0.1:60626 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:49:24.949998 10020 connection.cc:535] server connection from 127.0.0.1:37850 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:49:24.950235 10154 connection.cc:535] server connection from 127.0.0.1:34082 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:49:24.954099 10137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.942528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.3.0.252:39847)
0928 21:49:24.942566 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.942568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.947354 (+ 4786us) server_negotiation.cc:407] Connection header received
0928 21:49:24.947420 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.947421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.947439 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.947482 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.947990 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.948538 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.949226 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.949436 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.949490 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.950815 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.950817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.950817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.950824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.952431 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.952873 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.952875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.952875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.952930 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.953127 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.953128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.953128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.953256 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.953339 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.953910 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.953984 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:49:24.959153 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.952003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.3.0.253:38339)
0928 21:49:24.952064 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.952065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.952073 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:24.952093 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.952094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.952105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.952164 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.952646 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.953195 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.954450 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.954760 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.954830 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.956262 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.956264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.956264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.956271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.957923 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.958410 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.958411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.958412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.958469 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.958629 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.958629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.958630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.958752 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.958826 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.958988 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.959061 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:24.960333 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:24.953495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.3.0.252:39847)
0928 21:49:24.953530 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:24.953532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:24.954399 (+ 867us) server_negotiation.cc:407] Connection header received
0928 21:49:24.954417 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:24.954418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:24.954430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:24.954492 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:24.955107 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.955705 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.956499 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:24.956805 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:24.956873 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:24.957972 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:24.957973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:24.957974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:24.957985 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:24.959398 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.959676 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.959677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.959677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.959731 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:24.959882 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:24.959882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:24.959883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:24.960007 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:24.960099 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:24.960159 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 21:49:24.960233 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:49:25.938594 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:25.938290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:42539)
0928 21:49:25.938415 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:25.938536 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
W20260928 21:49:25.939018 10131 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:46579 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:25.939224 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:25.938933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:55067)
0928 21:49:25.939055 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:25.939179 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":75,"threads_started":1}
W20260928 21:49:25.939399 10404 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:46579 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:25.939494 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:25.939250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:35035)
0928 21:49:25.939385 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:25.939453 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":47,"threads_started":1}
W20260928 21:49:25.939659 10268 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:46579 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:26.442127 10463 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06fcd49e1d8745739eb7a281ab96ea8e)
I20260928 21:49:26.442291 10463 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:26.442520 10463 leader_election.cc:288] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579)
I20260928 21:49:26.442865 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:26.442672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:60046)
0928 21:49:26.442779 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:26.442827 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
W20260928 21:49:26.442998 9748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:26.443384 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:26.443273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:60052)
0928 21:49:26.443337 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:26.443365 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 21:49:26.443482 9748 leader_election.cc:334] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579): Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:26.443547 9748 leader_election.cc:302] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f2baa60ebbbf48aa8aa6636e8a5f6587; no voters: 06fcd49e1d8745739eb7a281ab96ea8e
I20260928 21:49:26.443643 10463 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:49:26.939602 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:26.939341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:60471)
0928 21:49:26.939505 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:26.939547 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260928 21:49:26.939927 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:26.939690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:59855)
0928 21:49:26.939840 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:26.939884 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260928 21:49:26.940232 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:26.940012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:41097)
0928 21:49:26.940156 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:26.940192 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":57,"threads_started":1}
I20260928 21:49:27.940439 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:27.940164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:51343)
0928 21:49:27.940316 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:27.940382 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260928 21:49:27.940727 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:27.940373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:55171)
0928 21:49:27.940650 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:27.940684 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":45,"threads_started":1}
I20260928 21:49:27.940908 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:27.940674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:54727)
0928 21:49:27.940835 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:27.940869 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":53,"threads_started":1}
I20260928 21:49:28.276846 10472 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06fcd49e1d8745739eb7a281ab96ea8e)
I20260928 21:49:28.276937 10472 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:28.277060 10472 leader_election.cc:288] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579)
I20260928 21:49:28.277323 10473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:28.277170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:60310)
0928 21:49:28.277257 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:28.277286 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":45,"threads_started":1}
I20260928 21:49:28.277676 10473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:28.277592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:60322)
0928 21:49:28.277634 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:28.277659 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:49:28.277755 9748 leader_election.cc:334] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579): Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:28.277803 9748 leader_election.cc:302] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f2baa60ebbbf48aa8aa6636e8a5f6587; no voters: 06fcd49e1d8745739eb7a281ab96ea8e
I20260928 21:49:28.277871 10472 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:49:28.941439 10475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:28.941213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:35503)
0928 21:49:28.941345 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:28.941384 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260928 21:49:28.941723 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:28.941532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:59211)
0928 21:49:28.941645 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:28.941677 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":43,"threads_started":1}
I20260928 21:49:28.941747 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:28.941376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:54831)
0928 21:49:28.941670 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:28.941706 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":51,"threads_started":1}
I20260928 21:49:29.942219 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:29.941993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:55273)
0928 21:49:29.942125 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:29.942165 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260928 21:49:29.942283 10479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:29.942079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:50981)
0928 21:49:29.942201 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:29.942234 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":47,"threads_started":1}
I20260928 21:49:29.942461 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:29.942073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:55035)
0928 21:49:29.942387 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:29.942422 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":55,"threads_started":1}
I20260928 21:49:29.949265 10481 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06fcd49e1d8745739eb7a281ab96ea8e)
I20260928 21:49:29.949335 10481 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:29.949457 10481 leader_election.cc:288] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579)
I20260928 21:49:29.949755 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:29.949570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:60496)
0928 21:49:29.949691 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:29.949724 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260928 21:49:29.950107 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:29.950014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:60500)
0928 21:49:29.950060 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:29.950088 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 21:49:29.950170 9748 leader_election.cc:334] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579): Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:29.950222 9748 leader_election.cc:302] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f2baa60ebbbf48aa8aa6636e8a5f6587; no voters: 06fcd49e1d8745739eb7a281ab96ea8e
I20260928 21:49:29.950290 10481 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:49:30.942994 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:30.942758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:55637)
0928 21:49:30.942877 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:30.942916 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260928 21:49:30.943109 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:30.942839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:38511)
0928 21:49:30.943015 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:30.943062 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260928 21:49:30.943149 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:30.942758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:36387)
0928 21:49:30.943077 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:30.943112 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":60,"threads_started":1}
I20260928 21:49:31.943704 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:31.943473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:36807)
0928 21:49:31.943607 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:31.943648 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":147,"threads_started":1}
I20260928 21:49:31.943743 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:31.943591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:58003)
0928 21:49:31.943684 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:31.943708 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
I20260928 21:49:31.943900 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:31.943456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:44215)
0928 21:49:31.943801 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:31.943852 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":63,"threads_started":1}
I20260928 21:49:32.944494 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:32.944244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:44919)
0928 21:49:32.944359 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:32.944413 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20260928 21:49:32.944573 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:32.944212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:36529)
0928 21:49:32.944501 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:32.944534 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":76,"threads_started":1}
I20260928 21:49:32.944625 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:32.944256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:34545)
0928 21:49:32.944540 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:32.944576 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":42,"threads_started":1}
I20260928 21:49:33.287400 10493 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06fcd49e1d8745739eb7a281ab96ea8e)
I20260928 21:49:33.287520 10493 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:33.287670 10493 leader_election.cc:288] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579)
I20260928 21:49:33.288002 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:33.287832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:46780)
0928 21:49:33.287928 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:33.287962 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":44,"threads_started":1}
W20260928 21:49:33.288087 9748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260928 21:49:33.288434 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:33.288348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:46784)
0928 21:49:33.288393 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:33.288416 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 21:49:33.288520 9748 leader_election.cc:334] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579): Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:33.288552 9748 leader_election.cc:302] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f2baa60ebbbf48aa8aa6636e8a5f6587; no voters: 06fcd49e1d8745739eb7a281ab96ea8e
I20260928 21:49:33.288632 10493 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:49:33.945194 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:33.944968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:55167)
0928 21:49:33.945096 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:33.945138 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260928 21:49:33.945273 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:33.944862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:40369)
0928 21:49:33.945189 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:33.945227 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":88,"threads_started":1}
I20260928 21:49:33.945349 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:33.944926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:49239)
0928 21:49:33.945278 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:33.945311 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":41,"threads_started":1}
I20260928 21:49:34.946031 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:34.945816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:50023)
0928 21:49:34.945935 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:34.945975 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260928 21:49:34.946231 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:34.945788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:58595)
0928 21:49:34.946148 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:34.946184 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":59,"threads_started":1}
I20260928 21:49:34.946355 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:34.946012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:56853)
0928 21:49:34.946265 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:34.946306 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":59,"threads_started":1}
I20260928 21:49:35.946856 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:35.946616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:38241)
0928 21:49:35.946759 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:35.946805 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":46,"threads_started":1}
I20260928 21:49:35.946856 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:35.946593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:37347)
0928 21:49:35.946763 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:35.946800 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
I20260928 21:49:35.947012 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:35.946761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:37259)
0928 21:49:35.946860 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:35.946886 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260928 21:49:36.947440 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:36.947232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:44865)
0928 21:49:36.947340 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:36.947383 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260928 21:49:36.947530 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:36.947336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:59599)
0928 21:49:36.947462 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:36.947494 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":46,"threads_started":1}
I20260928 21:49:36.947530 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:36.947211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:58009)
0928 21:49:36.947443 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:36.947482 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":60,"threads_started":1}
I20260928 21:49:37.948218 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:37.947958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:35577)
0928 21:49:37.948063 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:37.948168 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":55,"threads_started":1}
I20260928 21:49:37.948211 10508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:37.947920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:48893)
0928 21:49:37.948117 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:37.948157 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":60,"threads_started":1}
I20260928 21:49:37.948336 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:37.948146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:42751)
0928 21:49:37.948235 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:37.948261 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":41,"threads_started":1}
I20260928 21:49:38.518847 10511 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06fcd49e1d8745739eb7a281ab96ea8e)
I20260928 21:49:38.518946 10511 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:38.519090 10511 leader_election.cc:288] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579)
I20260928 21:49:38.519399 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:38.519228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:47096)
0928 21:49:38.519324 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:38.519361 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
W20260928 21:49:38.519479 9748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:38.519799 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:38.519714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:47102)
0928 21:49:38.519757 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:38.519782 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 21:49:38.519872 9748 leader_election.cc:334] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579): Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:38.519919 9748 leader_election.cc:302] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f2baa60ebbbf48aa8aa6636e8a5f6587; no voters: 06fcd49e1d8745739eb7a281ab96ea8e
I20260928 21:49:38.520005 10511 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:49:38.948931 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:38.948708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:55619)
0928 21:49:38.948865 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:38.948893 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":56,"threads_started":1}
I20260928 21:49:38.948931 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:38.948684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:52851)
0928 21:49:38.948838 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:38.948878 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260928 21:49:38.948954 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:38.948758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:37373)
0928 21:49:38.948882 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:38.948916 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":52,"threads_started":1}
I20260928 21:49:39.949645 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:39.949445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:53191)
0928 21:49:39.949550 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:39.949593 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":40,"threads_started":1}
I20260928 21:49:39.949645 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:39.949409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:48011)
0928 21:49:39.949554 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:39.949593 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":58,"threads_started":1}
I20260928 21:49:39.949645 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:39.949354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:46689)
0928 21:49:39.949568 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:39.949598 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
I20260928 21:49:40.950232 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:40.950019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:58079)
0928 21:49:40.950134 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:40.950176 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":40,"threads_started":1}
I20260928 21:49:40.950402 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:40.949988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:37869)
0928 21:49:40.950329 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:40.950362 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":72,"threads_started":1}
I20260928 21:49:40.950482 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:40.949987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:33201)
0928 21:49:40.950414 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:40.950444 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":68,"threads_started":1}
I20260928 21:49:41.398896 10523 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06fcd49e1d8745739eb7a281ab96ea8e)
I20260928 21:49:41.398993 10523 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:41.399138 10523 leader_election.cc:288] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579)
I20260928 21:49:41.399457 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:41.399292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:47238)
0928 21:49:41.399384 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:41.399417 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":43,"threads_started":1}
I20260928 21:49:41.399830 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:41.399744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:47252)
0928 21:49:41.399786 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:41.399811 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 21:49:41.399894 9748 leader_election.cc:334] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579): Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:41.399932 9748 leader_election.cc:302] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f2baa60ebbbf48aa8aa6636e8a5f6587; no voters: 06fcd49e1d8745739eb7a281ab96ea8e
I20260928 21:49:41.399987 10523 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:49:41.951148 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:41.950910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:57629)
0928 21:49:41.951046 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:41.951090 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260928 21:49:41.951149 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:41.950921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:50765)
0928 21:49:41.951065 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:41.951100 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260928 21:49:41.951342 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:41.950938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:45215)
0928 21:49:41.951273 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:41.951305 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":52,"threads_started":1}
I20260928 21:49:42.951956 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:42.951688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:46973)
0928 21:49:42.951889 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:42.951918 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":56,"threads_started":1}
I20260928 21:49:42.951917 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:42.951584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:36635)
0928 21:49:42.951788 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:42.951831 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260928 21:49:42.952157 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:42.951930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:38833)
0928 21:49:42.952073 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:42.952111 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":48,"threads_started":1}
I20260928 21:49:43.952612 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:43.952356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:51869)
0928 21:49:43.952509 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:43.952554 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260928 21:49:43.952751 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:43.952358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:56725)
0928 21:49:43.952632 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:43.952672 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":63,"threads_started":1}
I20260928 21:49:43.952968 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:43.952620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:44337)
0928 21:49:43.952896 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:43.952929 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":59,"threads_started":1}
I20260928 21:49:44.953511 10535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:44.953249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:35317)
0928 21:49:44.953415 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:44.953456 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":64,"threads_started":1}
I20260928 21:49:44.953699 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:44.953509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:53559)
0928 21:49:44.953613 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:44.953651 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":46,"threads_started":1}
I20260928 21:49:44.953974 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:44.953262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:60605)
0928 21:49:44.953867 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:44.953933 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":54,"threads_started":1}
I20260928 21:49:45.437184 10538 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06fcd49e1d8745739eb7a281ab96ea8e)
I20260928 21:49:45.437283 10538 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:45.437434 10538 leader_election.cc:288] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579)
I20260928 21:49:45.437748 10539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:45.437569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:39876)
0928 21:49:45.437665 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:45.437708 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":48,"threads_started":1}
W20260928 21:49:45.437826 9748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260928 21:49:45.438155 10539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:45.438050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:39890)
0928 21:49:45.438112 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:45.438136 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 21:49:45.438213 9748 leader_election.cc:334] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579): Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:45.438257 9748 leader_election.cc:302] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f2baa60ebbbf48aa8aa6636e8a5f6587; no voters: 06fcd49e1d8745739eb7a281ab96ea8e
I20260928 21:49:45.438324 10538 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:49:45.954525 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:45.954265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:56227)
0928 21:49:45.954428 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:45.954468 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260928 21:49:45.954586 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:45.954123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:34345)
0928 21:49:45.954496 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:45.954535 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":82,"threads_started":1}
I20260928 21:49:45.954689 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:45.954337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:45815)
0928 21:49:45.954620 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:45.954651 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":46,"threads_started":1}
I20260928 21:49:46.955224 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:46.954959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:50629)
0928 21:49:46.955126 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:46.955167 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
I20260928 21:49:46.955227 10545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:46.954959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:46709)
0928 21:49:46.955111 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:46.955176 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260928 21:49:46.955377 10546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:46.955162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:43177)
0928 21:49:46.955260 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:46.955335 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":49,"threads_started":1}
I20260928 21:49:47.955920 10548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:47.955716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:45337)
0928 21:49:47.955834 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:47.955869 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260928 21:49:47.955920 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:47.955716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:57011)
0928 21:49:47.955834 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:47.955869 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260928 21:49:47.956101 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:47.955909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:49445)
0928 21:49:47.956021 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:47.956058 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260928 21:49:48.956578 10550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:48.956323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:47767)
0928 21:49:48.956480 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:48.956521 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260928 21:49:48.956619 10551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:48.956355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:44699)
0928 21:49:48.956535 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:48.956572 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":54,"threads_started":1}
I20260928 21:49:48.956666 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:48.956400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:48953)
0928 21:49:48.956601 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:48.956629 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":43,"threads_started":1}
I20260928 21:49:49.957181 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:49.956957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:47301)
0928 21:49:49.957081 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:49.957126 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260928 21:49:49.957294 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:49.957122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:46635)
0928 21:49:49.957230 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:49.957257 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":49,"threads_started":1}
I20260928 21:49:49.957289 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:49.957109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:38587)
0928 21:49:49.957208 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:49.957243 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":55,"threads_started":1}
I20260928 21:49:50.957929 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:50.957687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:32933)
0928 21:49:50.957842 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:50.957883 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260928 21:49:50.957921 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:50.957687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:43821)
0928 21:49:50.957826 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:50.957867 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260928 21:49:50.957937 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:50.957687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:40633)
0928 21:49:50.957876 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:50.957902 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260928 21:49:51.958508 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:51.958326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:52797)
0928 21:49:51.958427 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:51.958459 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":40,"threads_started":1}
I20260928 21:49:51.958544 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:51.958298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:57799)
0928 21:49:51.958423 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:51.958459 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260928 21:49:51.958587 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:51.958390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:49755)
0928 21:49:51.958513 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:51.958545 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260928 21:49:52.959149 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:52.958882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:34653)
0928 21:49:52.959088 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:52.959114 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":65,"threads_started":1}
I20260928 21:49:52.959146 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:52.958890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:36429)
0928 21:49:52.959046 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:52.959091 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":57,"threads_started":1}
I20260928 21:49:52.959172 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:52.958884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:33111)
0928 21:49:52.959088 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:52.959126 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":67,"threads_started":1}
I20260928 21:49:53.959849 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:53.959467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:45515)
0928 21:49:53.959749 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:53.959798 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":67,"threads_started":1}
I20260928 21:49:53.959849 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:53.959655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:32823)
0928 21:49:53.959765 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:53.959798 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260928 21:49:53.959870 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:53.959665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:59421)
0928 21:49:53.959785 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:53.959824 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":45,"threads_started":1}
I20260928 21:49:54.054279 10568 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 06fcd49e1d8745739eb7a281ab96ea8e)
I20260928 21:49:54.054391 10568 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "06fcd49e1d8745739eb7a281ab96ea8e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 46579 } } peers { permanent_uuid: "f2baa60ebbbf48aa8aa6636e8a5f6587" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 38339 } attrs { promote: false } } peers { permanent_uuid: "73e4afeb2e034cc1be7c21e90c025570" member_type: NON_VOTER last_known_addr { host: "127.3.0.252" port: 39847 } attrs { promote: true } }
I20260928 21:49:54.054838 10568 leader_election.cc:288] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579)
I20260928 21:49:54.054956 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:54.054745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:40398)
0928 21:49:54.054856 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:54.054906 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
W20260928 21:49:54.055143 9748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:54.055554 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:54.055464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.0.0.1:40414)
0928 21:49:54.055503 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:54.055531 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 21:49:54.055641 9748 leader_election.cc:334] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 06fcd49e1d8745739eb7a281ab96ea8e (127.3.0.254:46579): Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
I20260928 21:49:54.055706 9748 leader_election.cc:302] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f2baa60ebbbf48aa8aa6636e8a5f6587; no voters: 06fcd49e1d8745739eb7a281ab96ea8e
I20260928 21:49:54.055788 10568 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f2baa60ebbbf48aa8aa6636e8a5f6587 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:49:54.960608 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:54.960223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.193:34519)
0928 21:49:54.960394 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:54.960550 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260928 21:49:54.960750 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:54.960389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.195:40717)
0928 21:49:54.960618 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:54.960653 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":44,"threads_started":1}
I20260928 21:49:54.960947 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:54.960615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:46579 (local address 127.3.0.194:35991)
0928 21:49:54.960830 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:54.960901 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":45,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.0.254:46579,127.3.0.253:38339,127.3.0.252:39847, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.3.0.254:46579,127.3.0.253:38339,127.3.0.252:39847, num_attempts: 1)
I20260928 21:49:55.102569 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 9911
I20260928 21:49:55.106813 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 10138
I20260928 21:49:55.110523 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 10274
I20260928 21:49:55.114845 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 9630
I20260928 21:49:55.118428 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 9818
2026-09-28T21:49:55Z chronyd exiting
I20260928 21:49:55.133255 3075 test_util.cc:175] -----------------------------------------------
I20260928 21:49:55.133303 3075 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790632139349757-3075-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32781 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:55 dist-test-slave-jrck krb5kdc[10576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:55 dist-test-slave-jrck krb5kdc[10576](info): set up 2 sockets
Sep 28 21:49:55 dist-test-slave-jrck krb5kdc[10576](info): commencing operation
krb5kdc: starting...
W20260928 21:49:56.072992 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.254.keytab.
Entry for principal oryx/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.254.keytab.
I20260928 21:49:56.179248 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33381
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.254.keytab
--principal=oryx/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:33381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:56.258788 10589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:56.258972 10589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:56.258993 10589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:56.260109 10589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:56.260146 10589 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:49:56.260159 10589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:56.260169 10589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:56.260179 10589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:56.260190 10589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:56.261699 10589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:33381
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.254.keytab
--principal=oryx/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:56.261879 10589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:56.262043 10589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:56.266394 10597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:56.266418 10594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:56.266412 10595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:56.266620 10589 server_base.cc:1034] running on GCE node
I20260928 21:49:56.266834 10589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:56.267052 10589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:56.268179 10589 hybrid_clock.cc:648] HybridClock initialized: now 1790632196268166 us; error 29 us; skew 500 ppm
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:56.270457 10589 init.cc:377] Logged in from keytab as oryx/127.3.0.254@KRBTEST.COM (short username oryx)
I20260928 21:49:56.270862 10589 webserver.cc:460] Webserver started at http://127.3.0.254:44073/ using document root <none> and password file <none>
I20260928 21:49:56.271046 10589 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:56.271085 10589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:56.271179 10589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:56.272109 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "e1286be4222441558a1bfb2145802e5b"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "86f46a74ee4ab4a5cc2540eaa5a11521"
server_key_iv: "cc168a68dad2685c9f606fac21ec830e"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.272388 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "e1286be4222441558a1bfb2145802e5b"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "86f46a74ee4ab4a5cc2540eaa5a11521"
server_key_iv: "cc168a68dad2685c9f606fac21ec830e"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.273562 10589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:49:56.274135 10604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.274271 10589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:56.274336 10589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "e1286be4222441558a1bfb2145802e5b"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "86f46a74ee4ab4a5cc2540eaa5a11521"
server_key_iv: "cc168a68dad2685c9f606fac21ec830e"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.274384 10589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:56.291836 10589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:56.293488 10589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:56.293589 10589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:56.297777 10589 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:33381
I20260928 21:49:56.297832 10666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:33381 every 8 connection(s)
I20260928 21:49:56.298115 10589 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:49:56.298731 10667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:56.300905 10667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b: Bootstrap starting.
I20260928 21:49:56.301448 10667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:56.301714 10667 log.cc:824] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:56.302305 10667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b: No bootstrap required, opened a new log
I20260928 21:49:56.303735 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 10589
I20260928 21:49:56.303819 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:49:56.303988 3075 external_mini_cluster.cc:1428] Setting key acde405ec4609e8fe60f6ac08f8b3f0b
I20260928 21:49:56.304443 10667 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1286be4222441558a1bfb2145802e5b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33381 } }
I20260928 21:49:56.304631 10667 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:56.304667 10667 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1286be4222441558a1bfb2145802e5b, State: Initialized, Role: FOLLOWER
I20260928 21:49:56.304797 10667 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1286be4222441558a1bfb2145802e5b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33381 } }
I20260928 21:49:56.304873 10667 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:49:56.304908 10667 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:49:56.304952 10667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:56.305464 10667 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1286be4222441558a1bfb2145802e5b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33381 } }
I20260928 21:49:56.305548 10667 leader_election.cc:302] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1286be4222441558a1bfb2145802e5b; no voters:
I20260928 21:49:56.305743 10667 leader_election.cc:288] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:49:56.305786 10672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:56.305924 10672 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [term 1 LEADER]: Becoming Leader. State: Replica: e1286be4222441558a1bfb2145802e5b, State: Running, Role: LEADER
I20260928 21:49:56.305981 10667 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:49:56.306047 10672 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1286be4222441558a1bfb2145802e5b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33381 } }
I20260928 21:49:56.306380 10672 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1286be4222441558a1bfb2145802e5b. Latest consensus state: current_term: 1 leader_uuid: "e1286be4222441558a1bfb2145802e5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1286be4222441558a1bfb2145802e5b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33381 } } }
I20260928 21:49:56.306439 10674 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1286be4222441558a1bfb2145802e5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1286be4222441558a1bfb2145802e5b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33381 } } }
I20260928 21:49:56.306520 10672 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:56.306527 10674 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b [sys.catalog]: This master's current role is: LEADER
I20260928 21:49:56.306800 10681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:49:56.307494 10681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:49:56.309159 10681 catalog_manager.cc:1347] Generated new cluster ID: abaa0ae4874b4304b6c893dcd5acfe65
I20260928 21:49:56.309213 10681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:56.314585 10681 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:49:56.315269 10681 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:49:56.317413 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.304447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.3.0.254:33381)
0928 21:49:56.304654 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.304664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.304697 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:49:56.304817 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.304824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.304878 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.305016 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:56.305613 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.307076 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.308125 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.308634 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.308732 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.314405 (+ 5673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:56.314408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:56.314408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:56.314419 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:56.315980 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.316299 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.316302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.316303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.316389 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.316581 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.316581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.316582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.316792 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:56.316942 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.317022 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.317258 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260928 21:49:56.322043 10681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b: Generated new TSK 0
I20260928 21:49:56.322184 10681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab.
Entry for principal oryx/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab.
I20260928 21:49:56.351325 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33381
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab
--principal=oryx/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:56.451188 10693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:56.451385 10693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:56.451416 10693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:56.452569 10693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:56.452610 10693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:56.452636 10693 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:49:56.452661 10693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:49:56.454164 10693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab
--principal=oryx/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.0.254:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:56.454383 10693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:56.454583 10693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:56.459945 10701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:56.459957 10699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:56.459951 10698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:56.460233 10693 server_base.cc:1034] running on GCE node
I20260928 21:49:56.460404 10693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:56.460668 10693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:56.461825 10693 hybrid_clock.cc:648] HybridClock initialized: now 1790632196461809 us; error 29 us; skew 500 ppm
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:56.464016 10693 init.cc:377] Logged in from keytab as oryx/127.3.0.193@KRBTEST.COM (short username oryx)
I20260928 21:49:56.464390 10693 webserver.cc:460] Webserver started at http://127.3.0.193:44683/ using document root <none> and password file <none>
I20260928 21:49:56.464617 10693 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:56.464664 10693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:56.464810 10693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:56.465812 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "94067b03f8fd417cbd4b26ed3b235313"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "00bb7c4f9acef11f5e5f08c23c0ef032"
server_key_iv: "d9bfdb66556443f3522eeebec97c3105"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.466148 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "94067b03f8fd417cbd4b26ed3b235313"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "00bb7c4f9acef11f5e5f08c23c0ef032"
server_key_iv: "d9bfdb66556443f3522eeebec97c3105"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.467343 10693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:49:56.467993 10708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.468148 10693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:56.468226 10693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "94067b03f8fd417cbd4b26ed3b235313"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "00bb7c4f9acef11f5e5f08c23c0ef032"
server_key_iv: "d9bfdb66556443f3522eeebec97c3105"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.468289 10693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:56.481097 10693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:56.482998 10693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:56.483132 10693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:56.483601 10693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:56.483637 10693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.483670 10693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:56.483700 10693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.489809 10693 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:33195
I20260928 21:49:56.489884 10823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:33195 every 8 connection(s)
I20260928 21:49:56.490140 10693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:56.493000 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.484123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:51341 (local address 127.3.0.254:33381)
0928 21:49:56.484174 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.484176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.484376 (+ 200us) server_negotiation.cc:407] Connection header received
0928 21:49:56.484927 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.484928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.484946 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.485009 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:56.485806 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.486571 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.488061 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.488397 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.488503 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.490911 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:56.490913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:56.490913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:56.490919 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:56.491953 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.492283 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.492283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.492284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.492326 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.492524 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.492524 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.492525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.492627 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:56.492674 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.492880 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.492939 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:49:56.493156 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.484115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33381 (local address 127.3.0.193:51341)
0928 21:49:56.484265 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.484299 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.484828 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.485093 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.485098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.485425 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:56.485693 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.485700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.486674 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.486675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.487962 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.487966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.488545 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.488547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.488698 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.489244 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:56.489254 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:56.490814 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:56.492015 (+ 1201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.492021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.492022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.492227 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.492369 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.492369 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.492370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.492474 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.492670 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:56.492674 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:56.492781 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.492853 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.492965 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260928 21:49:56.493589 10824 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33381
I20260928 21:49:56.493690 10824 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:56.493875 10824 heartbeater.cc:511] Master 127.3.0.254:33381 requested a full tablet report, sending...
I20260928 21:49:56.494410 10620 ts_manager.cc:194] Registered new tserver with Master: 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195)
I20260928 21:49:56.495209 10620 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:51341
I20260928 21:49:56.496397 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 10693
I20260928 21:49:56.496521 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:49:56.496727 3075 external_mini_cluster.cc:1428] Setting key 2a915665b0e4db35747522e81624da18
I20260928 21:49:56.499825 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.494686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:60117 (local address 127.3.0.254:33381)
0928 21:49:56.494723 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.494725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.494734 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:49:56.494753 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.494754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.494763 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.494821 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:56.495054 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.495674 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.496541 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.496765 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.496822 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.497597 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:56.497599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:56.497599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:56.497605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:56.498927 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.499229 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.499231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.499231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.499290 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.499434 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.499434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.499435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.499537 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:56.499590 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.499721 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.499771 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:49:56.499922 10830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.494526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33381 (local address 127.3.0.193:60117)
0928 21:49:56.494654 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.494665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.494717 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.494824 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.494825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.494950 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:56.495002 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.495003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.495760 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.495761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.496443 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.496446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.496823 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.496824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.496891 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.497205 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:56.497209 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:56.497528 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:56.498998 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.498998 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.498999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.499167 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.499347 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.499347 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.499348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.499388 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.499589 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:56.499590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:56.499639 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.499700 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.499787 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for oryx/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab.
Entry for principal oryx/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab.
I20260928 21:49:56.517282 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33381
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab
--principal=oryx/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:56.595160 10833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:56.595358 10833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:56.595381 10833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:56.596558 10833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:56.596599 10833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:56.596621 10833 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:49:56.596649 10833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:49:56.598137 10833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab
--principal=oryx/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.0.254:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:56.598357 10833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:56.598562 10833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:56.602993 10839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:56.603019 10833 server_base.cc:1034] running on GCE node
W20260928 21:49:56.602989 10841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:56.602990 10838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:56.603318 10833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:56.603526 10833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:56.604651 10833 hybrid_clock.cc:648] HybridClock initialized: now 1790632196604638 us; error 30 us; skew 500 ppm
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:56.606868 10833 init.cc:377] Logged in from keytab as oryx/127.3.0.194@KRBTEST.COM (short username oryx)
I20260928 21:49:56.607296 10833 webserver.cc:460] Webserver started at http://127.3.0.194:44823/ using document root <none> and password file <none>
I20260928 21:49:56.607494 10833 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:56.607535 10833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:56.607650 10833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:56.608661 10833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "89e3ac5da4f240a39b32ffa6673277c2"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "379650abafe3458508600e13519181cd"
server_key_iv: "3ba53ad1fa25629389a7074e0a27f7fa"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.608978 10833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "89e3ac5da4f240a39b32ffa6673277c2"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "379650abafe3458508600e13519181cd"
server_key_iv: "3ba53ad1fa25629389a7074e0a27f7fa"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.610109 10833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:56.610787 10848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.610911 10833 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:56.610973 10833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "89e3ac5da4f240a39b32ffa6673277c2"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "379650abafe3458508600e13519181cd"
server_key_iv: "3ba53ad1fa25629389a7074e0a27f7fa"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.611034 10833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:56.625603 10833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:56.627202 10833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:56.627317 10833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:56.627794 10833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:56.627835 10833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.627862 10833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:56.627893 10833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:56.634558 10833 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:38515
I20260928 21:49:56.634636 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:38515 every 8 connection(s)
I20260928 21:49:56.634994 10833 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:49:56.636546 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.628130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:49811 (local address 127.3.0.254:33381)
0928 21:49:56.628193 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.628195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.628395 (+ 200us) server_negotiation.cc:407] Connection header received
0928 21:49:56.628907 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.628909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.628923 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.628995 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:56.629595 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.630340 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.631416 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.631668 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.631727 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.634083 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:56.634085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:56.634085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:56.634093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:56.635551 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.635868 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.635868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.635869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.635913 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.636060 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.636060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.636060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.636147 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:56.636190 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.636377 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.636435 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:49:56.636688 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.628134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33381 (local address 127.3.0.194:49811)
0928 21:49:56.628273 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.628309 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.628812 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.629061 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.629065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.629315 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:56.629503 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.629507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.630427 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.630428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.631337 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.631340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.631734 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.631735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.631861 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.632365 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:56.632376 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:56.634009 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:56.635616 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.635620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.635621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.635813 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.635946 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.635946 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.635946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.636021 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.636188 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:56.636191 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:56.636291 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.636365 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.636501 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260928 21:49:56.637125 10964 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33381
I20260928 21:49:56.637199 10964 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:56.637364 10964 heartbeater.cc:511] Master 127.3.0.254:33381 requested a full tablet report, sending...
I20260928 21:49:56.637701 10621 ts_manager.cc:194] Registered new tserver with Master: 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515)
I20260928 21:49:56.638062 10621 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:49811
I20260928 21:49:56.641134 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 10833
I20260928 21:49:56.641224 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:49:56.641438 3075 external_mini_cluster.cc:1428] Setting key 1dbc7a8185c96faf224a24397bbbabe7
I20260928 21:49:56.642966 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.638123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:55375 (local address 127.3.0.254:33381)
0928 21:49:56.638158 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.638160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.638261 (+ 101us) server_negotiation.cc:407] Connection header received
0928 21:49:56.638319 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.638319 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.638328 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.638368 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:56.638709 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.639150 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.639798 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.639986 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.640037 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.640770 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:56.640771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:56.640771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:56.640778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:56.642018 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.642300 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.642301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.642301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.642346 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.642496 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.642497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.642497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.642623 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:56.642680 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.642810 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.642877 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:49:56.642976 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.638047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33381 (local address 127.3.0.194:55375)
0928 21:49:56.638210 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.638221 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.638271 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.638371 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.638372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.638534 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:56.638598 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.638599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.639240 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.639241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.639731 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.639732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.640039 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.640040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.640098 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.640412 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:56.640416 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:56.640699 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:56.642083 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.642083 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.642084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.642243 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.642395 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.642395 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.642395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.642448 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.642678 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:56.642679 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:56.642719 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.642782 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.642877 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":54,"threads_started":1}
WARNING: no policy specified for oryx/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab.
Entry for principal oryx/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab.
I20260928 21:49:56.662299 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:33381
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab
--principal=oryx/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:56.739089 10973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:56.739287 10973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:56.739310 10973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:56.740417 10973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:56.740474 10973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:56.740526 10973 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:49:56.740563 10973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:49:56.742015 10973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab
--principal=oryx/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.0.254:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:56.742218 10973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:56.742414 10973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:56.746987 10978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:56.746978 10981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:56.746989 10979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:56.747015 10973 server_base.cc:1034] running on GCE node
I20260928 21:49:56.747385 10973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:56.747609 10973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:56.748744 10973 hybrid_clock.cc:648] HybridClock initialized: now 1790632196748733 us; error 28 us; skew 500 ppm
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:49:56.751008 10973 init.cc:377] Logged in from keytab as oryx/127.3.0.195@KRBTEST.COM (short username oryx)
I20260928 21:49:56.751425 10973 webserver.cc:460] Webserver started at http://127.3.0.195:39081/ using document root <none> and password file <none>
I20260928 21:49:56.751618 10973 fs_manager.cc:359] Metadata directory not provided
I20260928 21:49:56.751663 10973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:49:56.751783 10973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:49:56.752820 10973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "77a318f5231747a4a139594fdb65d288"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "7b27036665d4cf3d7ed270804d5ed89a"
server_key_iv: "5b0f2a1767ae241bc8adc00dcb953d93"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.753145 10973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "77a318f5231747a4a139594fdb65d288"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "7b27036665d4cf3d7ed270804d5ed89a"
server_key_iv: "5b0f2a1767ae241bc8adc00dcb953d93"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.754290 10973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:49:56.754930 10988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.755069 10973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:56.755146 10973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "77a318f5231747a4a139594fdb65d288"
format_stamp: "Formatted at 2026-09-28 21:49:56 on dist-test-slave-jrck"
server_key: "7b27036665d4cf3d7ed270804d5ed89a"
server_key_iv: "5b0f2a1767ae241bc8adc00dcb953d93"
server_key_version: "encryptionkey@0"
I20260928 21:49:56.755201 10973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:49:56.773701 10973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:49:56.775746 10973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:49:56.775878 10973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:49:56.776372 10973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:49:56.776412 10973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.776440 10973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:49:56.776479 10973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:56.783252 10973 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:35949
I20260928 21:49:56.783324 11103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:35949 every 8 connection(s)
I20260928 21:49:56.783644 10973 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:49:56.785822 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.776806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:51973 (local address 127.3.0.254:33381)
0928 21:49:56.776872 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.776875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.777019 (+ 144us) server_negotiation.cc:407] Connection header received
0928 21:49:56.777521 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.777522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.777535 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.777604 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:56.778561 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.779298 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.780775 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.781074 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.781142 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.783208 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:56.783209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:56.783210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:56.783217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:56.784593 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.784952 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.784953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.784954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.785011 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.785223 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.785224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.785224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.785339 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:56.785395 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.785651 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.785732 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:49:56.785984 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.776761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33381 (local address 127.3.0.195:51973)
0928 21:49:56.776901 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.776939 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.777434 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.777740 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.777745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.778142 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:56.778421 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.778428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.779403 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.779405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.780700 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.780703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.781209 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.781209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.781350 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.781787 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:56.781795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:56.783101 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:56.784657 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.784664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.784666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.784885 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.785063 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.785063 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.785064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.785165 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.785392 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:56.785398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:56.785532 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.785619 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.785778 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260928 21:49:56.786237 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 10973
I20260928 21:49:56.786350 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:49:56.786544 11104 heartbeater.cc:347] Connected to a master server at 127.3.0.254:33381
I20260928 21:49:56.786566 3075 external_mini_cluster.cc:1428] Setting key 510d294c4ffee51754f85aaa6774f2b0
I20260928 21:49:56.786661 11104 heartbeater.cc:464] Registering TS with master...
I20260928 21:49:56.787025 11104 heartbeater.cc:511] Master 127.3.0.254:33381 requested a full tablet report, sending...
I20260928 21:49:56.787559 10620 ts_manager.cc:194] Registered new tserver with Master: 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949)
I20260928 21:49:56.787950 10620 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:51973
I20260928 21:49:56.791893 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.787440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:51185 (local address 127.3.0.254:33381)
0928 21:49:56.787472 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.787474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.787563 (+ 89us) server_negotiation.cc:407] Connection header received
0928 21:49:56.787607 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.787608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.787616 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.787670 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:56.787907 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.788555 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.789223 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.789398 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.789447 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.790194 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:56.790195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:56.790195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:56.790200 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:56.791145 (+ 945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.791405 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.791406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.791406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.791446 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.791564 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.791564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.791565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.791644 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:56.791685 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.791785 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.791835 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:49:56.791905 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.787408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33381 (local address 127.3.0.195:51185)
0928 21:49:56.787516 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.787526 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.787575 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.787673 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.787674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.787800 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:56.787856 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.787857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.788639 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.788640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.789158 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.789160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.789451 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.789452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.789497 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.789858 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:56.789862 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:56.790136 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:56.791203 (+ 1067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.791204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.791204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.791352 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.791487 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:56.791488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:56.791488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:56.791532 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:56.791681 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:56.791682 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:56.791725 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.791770 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.791838 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":67,"threads_started":1}
I20260928 21:49:56.798188 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:49:56.817621 10671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:49:56.812656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.3.0.254:33381)
0928 21:49:56.812715 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.812717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.812730 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:49:56.812790 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.812790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.812799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.812845 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:56.813334 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.813892 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.814618 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.814848 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.814896 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.817537 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60898: BlockingRecv error: recv got EOF from 127.0.0.1:60898 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 21:49:56 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:56.824592 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.818405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.3.0.254:33381)
0928 21:49:56.818459 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.818461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.818514 (+ 53us) server_negotiation.cc:407] Connection header received
0928 21:49:56.818574 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.818575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.818584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.818633 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:56.819049 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.819511 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.820153 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.820342 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.820394 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.822375 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:56.822377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:56.822377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:56.822383 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:56.823601 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.823904 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.823905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.823906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.823964 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:56.824127 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:56.824128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:56.824128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:56.824226 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:56.824309 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.824402 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.824492 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:49:56.825924 10620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:56.827431 10620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:56.833375 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.829465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:35949 (local address 127.0.0.1:34734)
0928 21:49:56.829893 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.829902 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.829959 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.830182 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.830183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.830185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:56.830257 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.830259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.831103 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.831104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.832299 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.832301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.833073 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.833074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.833169 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.833184 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.833254 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.833304 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260928 21:49:56.833420 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.829558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.3.0.195:35949)
0928 21:49:56.829745 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.829750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.829956 (+ 206us) server_negotiation.cc:407] Connection header received
0928 21:49:56.830022 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.830024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.830041 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.830111 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:56.830328 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.831031 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.832378 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.833005 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.833063 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.833139 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.833293 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.833329 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260928 21:49:56.833499 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.829652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:38515 (local address 127.0.0.1:38442)
0928 21:49:56.830015 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.830023 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.830068 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.830343 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.830344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.830345 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:56.830377 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.830378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.831552 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.831553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.832354 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.832356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.833282 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.833283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.833344 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.833348 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.833393 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.833432 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":60,"threads_started":1}
I20260928 21:49:56.833712 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.829746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.3.0.194:38515)
0928 21:49:56.829887 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.829891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.830068 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:49:56.830130 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.830132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.830145 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.830283 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:56.830444 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.831476 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.832426 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.833116 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.833180 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.833248 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.833577 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.833623 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260928 21:49:56.833823 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.829465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:33195 (local address 127.0.0.1:50660)
0928 21:49:56.829800 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.829838 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.829905 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.830176 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.830179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.830181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:56.830244 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.830246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.831536 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.831536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.832608 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.832611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.833575 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.833575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.833640 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.833643 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.833691 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.833725 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":27,"thread_start_us":58,"threads_started":1}
I20260928 21:49:56.834106 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.829560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.3.0.193:33195)
0928 21:49:56.829745 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.829750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.829903 (+ 153us) server_negotiation.cc:407] Connection header received
0928 21:49:56.830003 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.830007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.830023 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.830095 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:56.830318 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.831453 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.832689 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.833423 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.833477 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.833543 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.833964 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.834003 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260928 21:49:56.834339 11027 tablet_service.cc:1467] Processing CreateTablet for tablet 1162ad9e368f4265a3c9b0daf3bb3769 (DEFAULT_TABLE table=test-table [id=daaa2375b6fb40b18600126a89dd22a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:56.834367 10898 tablet_service.cc:1467] Processing CreateTablet for tablet 1162ad9e368f4265a3c9b0daf3bb3769 (DEFAULT_TABLE table=test-table [id=daaa2375b6fb40b18600126a89dd22a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:56.834683 11027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1162ad9e368f4265a3c9b0daf3bb3769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:56.834699 10898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1162ad9e368f4265a3c9b0daf3bb3769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:56.837066 10758 tablet_service.cc:1467] Processing CreateTablet for tablet 1162ad9e368f4265a3c9b0daf3bb3769 (DEFAULT_TABLE table=test-table [id=daaa2375b6fb40b18600126a89dd22a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:56.837381 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1162ad9e368f4265a3c9b0daf3bb3769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:56.837620 11133 tablet_bootstrap.cc:492] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: Bootstrap starting.
I20260928 21:49:56.838183 11133 tablet_bootstrap.cc:654] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:56.838429 11133 log.cc:824] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:56.839061 11133 tablet_bootstrap.cc:492] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: No bootstrap required, opened a new log
I20260928 21:49:56.839111 11133 ts_tablet_manager.cc:1397] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:56.840253 11135 tablet_bootstrap.cc:492] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: Bootstrap starting.
I20260928 21:49:56.840435 11133 raft_consensus.cc:348] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } }
I20260928 21:49:56.840555 11132 tablet_bootstrap.cc:492] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: Bootstrap starting.
I20260928 21:49:56.840610 11133 raft_consensus.cc:374] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:56.840668 11133 raft_consensus.cc:729] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a318f5231747a4a139594fdb65d288, State: Initialized, Role: FOLLOWER
I20260928 21:49:56.840792 11133 consensus_queue.cc:260] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } }
I20260928 21:49:56.840942 11135 tablet_bootstrap.cc:654] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:56.841055 11104 heartbeater.cc:503] Master 127.3.0.254:33381 was elected leader, sending a full tablet report...
I20260928 21:49:56.841079 11132 tablet_bootstrap.cc:654] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:56.841058 11133 ts_tablet_manager.cc:1428] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:49:56.841305 11135 log.cc:824] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:56.841367 11132 log.cc:824] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:49:56.842059 11135 tablet_bootstrap.cc:492] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: No bootstrap required, opened a new log
I20260928 21:49:56.842155 11135 ts_tablet_manager.cc:1397] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:49:56.842329 11132 tablet_bootstrap.cc:492] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: No bootstrap required, opened a new log
I20260928 21:49:56.842381 11132 ts_tablet_manager.cc:1397] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:56.843627 11135 raft_consensus.cc:348] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } }
I20260928 21:49:56.843753 11135 raft_consensus.cc:374] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:56.843783 11135 raft_consensus.cc:729] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94067b03f8fd417cbd4b26ed3b235313, State: Initialized, Role: FOLLOWER
I20260928 21:49:56.843887 11135 consensus_queue.cc:260] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } }
I20260928 21:49:56.843858 11132 raft_consensus.cc:348] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } }
I20260928 21:49:56.843981 11132 raft_consensus.cc:374] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:56.844020 11132 raft_consensus.cc:729] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e3ac5da4f240a39b32ffa6673277c2, State: Initialized, Role: FOLLOWER
I20260928 21:49:56.844121 11135 ts_tablet_manager.cc:1428] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:49:56.844123 11132 consensus_queue.cc:260] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } }
I20260928 21:49:56.844229 10824 heartbeater.cc:503] Master 127.3.0.254:33381 was elected leader, sending a full tablet report...
I20260928 21:49:56.844386 10964 heartbeater.cc:503] Master 127.3.0.254:33381 was elected leader, sending a full tablet report...
I20260928 21:49:56.844359 11132 ts_tablet_manager.cc:1428] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260928 21:49:56.885543 10965 tablet.cc:2369] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:49:56.888255 11136 raft_consensus.cc:482] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:56.888355 11136 raft_consensus.cc:504] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } }
I20260928 21:49:56.888706 11136 leader_election.cc:288] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195), 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515)
I20260928 21:49:56.891361 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.888960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:55063 (local address 127.3.0.194:38515)
0928 21:49:56.888997 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.888999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.889192 (+ 193us) server_negotiation.cc:407] Connection header received
0928 21:49:56.889226 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.889227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.889240 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.889290 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:56.889424 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.889875 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.890722 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.891061 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.891117 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.891128 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.891266 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.891300 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:49:56.891366 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.888930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:38447 (local address 127.3.0.193:33195)
0928 21:49:56.888990 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.888992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.889003 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:49:56.889051 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.889051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.889064 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.889123 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:56.889265 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.889811 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.890715 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.891063 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.891120 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.891129 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.891267 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.891302 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:49:56.891367 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.888874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:33195 (local address 127.3.0.195:38447)
0928 21:49:56.888933 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.888945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.888990 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.889131 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.889131 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.889133 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:56.889199 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.889201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.889882 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.889883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.890630 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.890632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.891132 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.891133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.891206 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.891208 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.891254 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.891293 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260928 21:49:56.891376 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.888881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:38515 (local address 127.3.0.195:55063)
0928 21:49:56.889108 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.889117 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.889180 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.889298 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.889298 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.889299 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:56.889368 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.889370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.889942 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.889943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.890638 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.890640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.891128 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.891129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.891202 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.891205 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.891250 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.891286 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":47,"thread_start_us":115,"threads_started":1}
I20260928 21:49:56.891723 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162ad9e368f4265a3c9b0daf3bb3769" candidate_uuid: "77a318f5231747a4a139594fdb65d288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94067b03f8fd417cbd4b26ed3b235313" is_pre_election: true
I20260928 21:49:56.891737 10918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162ad9e368f4265a3c9b0daf3bb3769" candidate_uuid: "77a318f5231747a4a139594fdb65d288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e3ac5da4f240a39b32ffa6673277c2" is_pre_election: true
I20260928 21:49:56.891870 10918 raft_consensus.cc:2454] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a318f5231747a4a139594fdb65d288 in term 0.
I20260928 21:49:56.891871 10778 raft_consensus.cc:2454] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a318f5231747a4a139594fdb65d288 in term 0.
I20260928 21:49:56.892114 10991 leader_election.cc:302] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a318f5231747a4a139594fdb65d288, 94067b03f8fd417cbd4b26ed3b235313; no voters:
I20260928 21:49:56.892246 11136 raft_consensus.cc:2781] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:56.892308 11136 raft_consensus.cc:482] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:56.892331 11136 raft_consensus.cc:3037] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:56.893210 11136 raft_consensus.cc:504] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } }
I20260928 21:49:56.893325 11136 leader_election.cc:288] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [CANDIDATE]: Term 1 election: Requested vote from peers 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195), 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515)
I20260928 21:49:56.893452 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162ad9e368f4265a3c9b0daf3bb3769" candidate_uuid: "77a318f5231747a4a139594fdb65d288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94067b03f8fd417cbd4b26ed3b235313"
I20260928 21:49:56.893468 10918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1162ad9e368f4265a3c9b0daf3bb3769" candidate_uuid: "77a318f5231747a4a139594fdb65d288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e3ac5da4f240a39b32ffa6673277c2"
I20260928 21:49:56.893530 10918 raft_consensus.cc:3037] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:56.893529 10778 raft_consensus.cc:3037] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:56.894342 10918 raft_consensus.cc:2454] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a318f5231747a4a139594fdb65d288 in term 1.
I20260928 21:49:56.894342 10778 raft_consensus.cc:2454] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a318f5231747a4a139594fdb65d288 in term 1.
I20260928 21:49:56.894497 10990 leader_election.cc:302] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a318f5231747a4a139594fdb65d288, 89e3ac5da4f240a39b32ffa6673277c2; no voters:
I20260928 21:49:56.894584 11136 raft_consensus.cc:2781] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:56.894723 11136 raft_consensus.cc:686] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 1 LEADER]: Becoming Leader. State: Replica: 77a318f5231747a4a139594fdb65d288, State: Running, Role: LEADER
I20260928 21:49:56.894815 11136 consensus_queue.cc:237] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } }
I20260928 21:49:56.895613 10620 catalog_manager.cc:5495] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77a318f5231747a4a139594fdb65d288 (127.3.0.195). New cstate: current_term: 1 leader_uuid: "77a318f5231747a4a139594fdb65d288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } health_report { overall_health: UNKNOWN } } }
I20260928 21:49:56.910548 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.908571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34748 (local address 127.3.0.195:35949)
0928 21:49:56.908609 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.908610 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.908620 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:49:56.908680 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.908681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.908697 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.908756 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:56.908866 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.909311 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.909986 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.910170 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.910213 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.910321 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:56.910413 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:56.910464 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.910482 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.910496 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:49:56.912180 10778 raft_consensus.cc:1261] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Refusing update from remote peer 77a318f5231747a4a139594fdb65d288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:56.912177 10918 raft_consensus.cc:1261] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Refusing update from remote peer 77a318f5231747a4a139594fdb65d288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:56.912511 11136 consensus_queue.cc:1035] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:56.912650 11142 consensus_queue.cc:1035] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:56.915019 11145 mvcc.cc:205] Tried to move back new op lower bound from 7334429478549872640 to 7334429478481809408. Current Snapshot: MvccSnapshot[applied={T|T < 7334429478549872640}]
I20260928 21:49:56.915695 11147 mvcc.cc:205] Tried to move back new op lower bound from 7334429478549872640 to 7334429478481809408. Current Snapshot: MvccSnapshot[applied={T|T < 7334429478549872640}]
I20260928 21:49:56.917096 10620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:56.917173 10620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:56.917850 10620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b: Sending DeleteTablet for 3 replicas of tablet 1162ad9e368f4265a3c9b0daf3bb3769
I20260928 21:49:56.918174 11027 tablet_service.cc:1514] Processing DeleteTablet for tablet 1162ad9e368f4265a3c9b0daf3bb3769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:56 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:34734
I20260928 21:49:56.918169 10758 tablet_service.cc:1514] Processing DeleteTablet for tablet 1162ad9e368f4265a3c9b0daf3bb3769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:56 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:50660
I20260928 21:49:56.918162 10898 tablet_service.cc:1514] Processing DeleteTablet for tablet 1162ad9e368f4265a3c9b0daf3bb3769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:56 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:38442
I20260928 21:49:56.918553 10620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:49:56.918675 11155 tablet_replica.cc:330] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: stopping tablet replica
W20260928 21:49:56.918710 10620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:56.918759 11155 raft_consensus.cc:2229] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:56.918874 11156 tablet_replica.cc:330] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: stopping tablet replica
I20260928 21:49:56.918984 11156 raft_consensus.cc:2229] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:56.919358 11156 raft_consensus.cc:2258] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:56.919548 11155 raft_consensus.cc:2258] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:56.919878 10758 tablet_service.cc:1467] Processing CreateTablet for tablet 91aea9af814544a2b80a2f9930491834 (DEFAULT_TABLE table=test-table [id=f910d14479c2424fa016104f4f0cc8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:56.919992 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91aea9af814544a2b80a2f9930491834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:56.919993 10898 tablet_service.cc:1467] Processing CreateTablet for tablet 91aea9af814544a2b80a2f9930491834 (DEFAULT_TABLE table=test-table [id=f910d14479c2424fa016104f4f0cc8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:56.920068 11155 ts_tablet_manager.cc:1905] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:56.920090 10898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91aea9af814544a2b80a2f9930491834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:56.920601 11027 tablet_service.cc:1467] Processing CreateTablet for tablet 91aea9af814544a2b80a2f9930491834 (DEFAULT_TABLE table=test-table [id=f910d14479c2424fa016104f4f0cc8f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:56.920718 11027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91aea9af814544a2b80a2f9930491834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:56.920714 11156 ts_tablet_manager.cc:1905] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:56.921200 11135 tablet_bootstrap.cc:492] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: Bootstrap starting.
I20260928 21:49:56.921582 11135 tablet_bootstrap.cc:654] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:56.921720 11155 ts_tablet_manager.cc:1918] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:56.921782 11155 log.cc:1196] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/1162ad9e368f4265a3c9b0daf3bb3769
I20260928 21:49:56.921926 11133 tablet_bootstrap.cc:492] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288: Bootstrap starting.
I20260928 21:49:56.922119 11156 ts_tablet_manager.cc:1918] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:56.922215 11156 log.cc:1196] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/1162ad9e368f4265a3c9b0daf3bb3769
I20260928 21:49:56.922312 11133 tablet_bootstrap.cc:654] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:56.922511 11156 ts_tablet_manager.cc:1939] T 1162ad9e368f4265a3c9b0daf3bb3769 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting consensus metadata
I20260928 21:49:56.922775 11155 ts_tablet_manager.cc:1939] T 1162ad9e368f4265a3c9b0daf3bb3769 P 94067b03f8fd417cbd4b26ed3b235313: Deleting consensus metadata
I20260928 21:49:56.922857 11157 tablet_replica.cc:330] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: stopping tablet replica
I20260928 21:49:56.922967 11157 raft_consensus.cc:2229] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:56.923066 11157 raft_consensus.cc:2258] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:56.923089 11132 tablet_bootstrap.cc:492] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2: Bootstrap starting.
I20260928 21:49:56.923144 10607 catalog_manager.cc:4841] TS 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195): tablet 1162ad9e368f4265a3c9b0daf3bb3769 (table test-table [id=daaa2375b6fb40b18600126a89dd22a1]) successfully deleted
I20260928 21:49:56.923434 11132 tablet_bootstrap.cc:654] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:56.923556 11157 ts_tablet_manager.cc:1905] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:56.923847 10606 catalog_manager.cc:4841] TS 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515): tablet 1162ad9e368f4265a3c9b0daf3bb3769 (table test-table [id=daaa2375b6fb40b18600126a89dd22a1]) successfully deleted
I20260928 21:49:56.923915 11132 tablet_bootstrap.cc:492] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2: No bootstrap required, opened a new log
I20260928 21:49:56.923947 11132 ts_tablet_manager.cc:1397] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:56.924079 11132 raft_consensus.cc:348] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:56.924139 11132 raft_consensus.cc:374] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:56.924157 11132 raft_consensus.cc:729] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e3ac5da4f240a39b32ffa6673277c2, State: Initialized, Role: FOLLOWER
I20260928 21:49:56.924208 11132 consensus_queue.cc:260] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:56.924289 11132 ts_tablet_manager.cc:1428] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.924432 11133 tablet_bootstrap.cc:492] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288: No bootstrap required, opened a new log
I20260928 21:49:56.924489 11133 ts_tablet_manager.cc:1397] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 21:49:56.924625 11133 raft_consensus.cc:348] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:56.924711 11133 raft_consensus.cc:374] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:56.924731 11133 raft_consensus.cc:729] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a318f5231747a4a139594fdb65d288, State: Initialized, Role: FOLLOWER
I20260928 21:49:56.924765 11133 consensus_queue.cc:260] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:56.924836 11133 ts_tablet_manager.cc:1428] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.924937 11157 ts_tablet_manager.cc:1918] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:56.925006 11157 log.cc:1196] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/1162ad9e368f4265a3c9b0daf3bb3769
I20260928 21:49:56.925226 11157 ts_tablet_manager.cc:1939] T 1162ad9e368f4265a3c9b0daf3bb3769 P 77a318f5231747a4a139594fdb65d288: Deleting consensus metadata
I20260928 21:49:56.925640 10606 catalog_manager.cc:4841] TS 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949): tablet 1162ad9e368f4265a3c9b0daf3bb3769 (table test-table [id=daaa2375b6fb40b18600126a89dd22a1]) successfully deleted
I20260928 21:49:56.928812 11135 tablet_bootstrap.cc:492] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: No bootstrap required, opened a new log
I20260928 21:49:56.928856 11135 ts_tablet_manager.cc:1397] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 21:49:56.928982 11135 raft_consensus.cc:348] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:56.929036 11135 raft_consensus.cc:374] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:56.929054 11135 raft_consensus.cc:729] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94067b03f8fd417cbd4b26ed3b235313, State: Initialized, Role: FOLLOWER
I20260928 21:49:56.929092 11135 consensus_queue.cc:260] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:56.929174 11135 ts_tablet_manager.cc:1428] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:56.974545 11139 raft_consensus.cc:482] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:56.974663 11139 raft_consensus.cc:504] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:56.975018 11139 leader_election.cc:288] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515), 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949)
I20260928 21:49:56.978307 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.975228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:47481 (local address 127.3.0.194:38515)
0928 21:49:56.975276 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.975278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.975288 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:49:56.975314 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.975315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.975325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.975399 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:56.975611 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.976344 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.977589 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.977990 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.978054 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.978065 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.978197 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.978234 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:49:56.978329 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.975150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:38515 (local address 127.3.0.193:47481)
0928 21:49:56.975202 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.975214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.975255 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.975455 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.975456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.975458 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:56.975528 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.975530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.976419 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.976420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.977505 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.977507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.978055 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.978056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.978140 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.978143 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.978186 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.978241 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1}
I20260928 21:49:56.978545 10918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91aea9af814544a2b80a2f9930491834" candidate_uuid: "94067b03f8fd417cbd4b26ed3b235313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e3ac5da4f240a39b32ffa6673277c2" is_pre_election: true
I20260928 21:49:56.978588 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.975376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:58577 (local address 127.3.0.195:35949)
0928 21:49:56.975416 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:56.975417 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:56.975425 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:56.975494 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:56.975495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:56.975504 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:56.975547 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:56.975684 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.976141 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.977585 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.978202 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.978269 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.978278 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:56.978485 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 21:49:56.978521 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:49:56.978628 10918 raft_consensus.cc:2454] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94067b03f8fd417cbd4b26ed3b235313 in term 0.
I20260928 21:49:56.978775 10710 leader_election.cc:302] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89e3ac5da4f240a39b32ffa6673277c2, 94067b03f8fd417cbd4b26ed3b235313; no voters:
I20260928 21:49:56.978837 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:56.975265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:35949 (local address 127.3.0.193:58577)
0928 21:49:56.975376 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:56.975384 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:56.975428 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:56.975555 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:56.975556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:56.975557 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:56.975631 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.975632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.976216 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.976217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.977481 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:56.977484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:56.978334 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:56.978335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:56.978407 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:56.978411 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:56.978663 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 21:49:56.978710 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260928 21:49:56.978919 11139 raft_consensus.cc:2781] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:56.978953 11139 raft_consensus.cc:482] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:56.978972 11139 raft_consensus.cc:3037] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:56.979065 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91aea9af814544a2b80a2f9930491834" candidate_uuid: "94067b03f8fd417cbd4b26ed3b235313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a318f5231747a4a139594fdb65d288" is_pre_election: true
I20260928 21:49:56.979183 11045 raft_consensus.cc:2454] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94067b03f8fd417cbd4b26ed3b235313 in term 0.
I20260928 21:49:56.979744 11139 raft_consensus.cc:504] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:56.979856 11139 leader_election.cc:288] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [CANDIDATE]: Term 1 election: Requested vote from peers 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515), 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949)
I20260928 21:49:56.980021 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91aea9af814544a2b80a2f9930491834" candidate_uuid: "94067b03f8fd417cbd4b26ed3b235313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a318f5231747a4a139594fdb65d288"
I20260928 21:49:56.980019 10918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91aea9af814544a2b80a2f9930491834" candidate_uuid: "94067b03f8fd417cbd4b26ed3b235313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e3ac5da4f240a39b32ffa6673277c2"
I20260928 21:49:56.980093 11045 raft_consensus.cc:3037] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:56.980093 10918 raft_consensus.cc:3037] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:56.980885 10918 raft_consensus.cc:2454] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94067b03f8fd417cbd4b26ed3b235313 in term 1.
I20260928 21:49:56.980885 11045 raft_consensus.cc:2454] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94067b03f8fd417cbd4b26ed3b235313 in term 1.
I20260928 21:49:56.981052 10710 leader_election.cc:302] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a318f5231747a4a139594fdb65d288, 94067b03f8fd417cbd4b26ed3b235313; no voters:
I20260928 21:49:56.981132 11139 raft_consensus.cc:2781] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:56.981245 11139 raft_consensus.cc:686] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 LEADER]: Becoming Leader. State: Replica: 94067b03f8fd417cbd4b26ed3b235313, State: Running, Role: LEADER
I20260928 21:49:56.981309 11139 consensus_queue.cc:237] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:56.981977 10620 catalog_manager.cc:5495] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193). New cstate: current_term: 1 leader_uuid: "94067b03f8fd417cbd4b26ed3b235313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } health_report { overall_health: UNKNOWN } } }
W20260928 21:49:56.990753 10825 tablet.cc:2369] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 21:49:57 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632196, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.254@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:57.028023 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.021088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:33381 (local address 127.0.0.1:60910)
0928 21:49:57.021239 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:57.021249 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:57.021308 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:57.021462 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:57.021462 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:57.021757 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:49:57.021806 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.021808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.022707 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.022707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.023219 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.023220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.023670 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.023670 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.023718 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.024184 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:49:57.024188 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:49:57.025651 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:49:57.027176 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:57.027183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:57.027184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:57.027365 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:57.027516 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:49:57.027517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:49:57.027517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:49:57.027563 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:49:57.027770 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:49:57.027776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:49:57.027815 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 21:49:57.027862 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:49:57.027950 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260928 21:49:57.028122 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.021163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.3.0.254:33381)
0928 21:49:57.021231 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.021233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.021311 (+ 78us) server_negotiation.cc:407] Connection header received
0928 21:49:57.021356 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.021357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.021371 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.021449 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:57.021873 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.022622 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.023311 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.023593 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.023667 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.025729 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:57.025730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:57.025731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:57.025738 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:57.027098 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:57.027418 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:57.027419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:57.027419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:57.027469 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:57.027605 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:57.027605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:57.027605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:57.027721 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:57.027777 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.027894 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.028004 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":36}
I20260928 21:49:57.029940 10620 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:60910:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:49:57.030148 10620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:57.031440 10758 tablet_service.cc:1467] Processing CreateTablet for tablet 1895c982ede04bceadf9c354170b263f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=548cd0163c1d44c991e6d0ad3b3c8710]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:49:57.031458 10898 tablet_service.cc:1467] Processing CreateTablet for tablet 1895c982ede04bceadf9c354170b263f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=548cd0163c1d44c991e6d0ad3b3c8710]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:49:57.031478 11027 tablet_service.cc:1467] Processing CreateTablet for tablet 1895c982ede04bceadf9c354170b263f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=548cd0163c1d44c991e6d0ad3b3c8710]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:49:57.031594 10898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1895c982ede04bceadf9c354170b263f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:57.031605 11027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1895c982ede04bceadf9c354170b263f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:57.031621 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1895c982ede04bceadf9c354170b263f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:57.032680 11133 tablet_bootstrap.cc:492] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: Bootstrap starting.
I20260928 21:49:57.032961 11133 tablet_bootstrap.cc:654] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:57.033070 11135 tablet_bootstrap.cc:492] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: Bootstrap starting.
I20260928 21:49:57.033423 11135 tablet_bootstrap.cc:654] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:57.033453 11133 tablet_bootstrap.cc:492] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: No bootstrap required, opened a new log
I20260928 21:49:57.033484 11133 ts_tablet_manager.cc:1397] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:49:57.033561 11133 raft_consensus.cc:348] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.033607 11133 raft_consensus.cc:374] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:57.033617 11133 raft_consensus.cc:729] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a318f5231747a4a139594fdb65d288, State: Initialized, Role: FOLLOWER
I20260928 21:49:57.033658 11133 consensus_queue.cc:260] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.033674 11132 tablet_bootstrap.cc:492] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: Bootstrap starting.
I20260928 21:49:57.033744 11133 ts_tablet_manager.cc:1428] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:57.033850 11152 tablet_replica.cc:439] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } }
I20260928 21:49:57.033897 11152 tablet_replica.cc:442] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:57.034029 11132 tablet_bootstrap.cc:654] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: Neither blocks nor log segments found. Creating new log.
W20260928 21:49:57.034235 11105 tablet.cc:2369] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:49:57.034578 11132 tablet_bootstrap.cc:492] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: No bootstrap required, opened a new log
I20260928 21:49:57.034624 11135 tablet_bootstrap.cc:492] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: No bootstrap required, opened a new log
I20260928 21:49:57.034628 11132 ts_tablet_manager.cc:1397] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:57.034662 11135 ts_tablet_manager.cc:1397] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:49:57.034801 11135 raft_consensus.cc:348] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.034796 11132 raft_consensus.cc:348] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.034863 11135 raft_consensus.cc:374] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:57.034863 11132 raft_consensus.cc:374] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:57.034889 11132 raft_consensus.cc:729] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e3ac5da4f240a39b32ffa6673277c2, State: Initialized, Role: FOLLOWER
I20260928 21:49:57.034889 11135 raft_consensus.cc:729] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94067b03f8fd417cbd4b26ed3b235313, State: Initialized, Role: FOLLOWER
I20260928 21:49:57.034933 11135 consensus_queue.cc:260] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.034933 11132 consensus_queue.cc:260] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.035017 11132 ts_tablet_manager.cc:1428] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:57.035022 11135 ts_tablet_manager.cc:1428] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:57.035074 11139 tablet_replica.cc:439] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } }
I20260928 21:49:57.035154 11139 tablet_replica.cc:442] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:57.035143 11140 tablet_replica.cc:439] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } }
I20260928 21:49:57.035224 11140 tablet_replica.cc:442] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:57.215579 11140 raft_consensus.cc:482] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:57.215682 11140 raft_consensus.cc:504] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.215969 11140 leader_election.cc:288] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195), 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949)
I20260928 21:49:57.218734 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.216199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:44883 (local address 127.3.0.195:35949)
0928 21:49:57.216274 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.216276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.216336 (+ 60us) server_negotiation.cc:407] Connection header received
0928 21:49:57.216399 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.216400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.216412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.216480 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:57.216608 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.217064 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.218075 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.218427 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.218486 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.218496 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.218627 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.218668 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:49:57.218753 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.216126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:35949 (local address 127.3.0.194:44883)
0928 21:49:57.216264 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:57.216275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:57.216344 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:57.216482 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:57.216482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:57.216484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:57.216552 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.216554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.217130 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.217131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.217997 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.218000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.218494 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.218494 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.218566 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.218570 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:57.218616 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:49:57.218660 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260928 21:49:57.218766 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.216212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:39843 (local address 127.3.0.193:33195)
0928 21:49:57.216274 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.216276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.216308 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:49:57.216382 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.216383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.216391 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.216438 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:49:57.216597 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.217065 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.218099 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.218454 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.218509 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.218519 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.218657 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.218694 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:49:57.218902 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.216126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:33195 (local address 127.3.0.194:39843)
0928 21:49:57.216254 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:57.216266 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:57.216325 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:57.216446 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:57.216447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:57.216449 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:49:57.216542 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.216544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.217133 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.217134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.218017 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.218019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.218527 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.218528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.218601 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.218605 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:49:57.218652 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:49:57.218693 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":44,"spinlock_wait_cycles":1408,"thread_start_us":78,"threads_started":1}
I20260928 21:49:57.218962 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1895c982ede04bceadf9c354170b263f" candidate_uuid: "89e3ac5da4f240a39b32ffa6673277c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a318f5231747a4a139594fdb65d288" is_pre_election: true
I20260928 21:49:57.219065 11045 raft_consensus.cc:2454] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e3ac5da4f240a39b32ffa6673277c2 in term 0.
I20260928 21:49:57.219125 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1895c982ede04bceadf9c354170b263f" candidate_uuid: "89e3ac5da4f240a39b32ffa6673277c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94067b03f8fd417cbd4b26ed3b235313" is_pre_election: true
I20260928 21:49:57.219213 10778 raft_consensus.cc:2454] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e3ac5da4f240a39b32ffa6673277c2 in term 0.
I20260928 21:49:57.219254 10850 leader_election.cc:302] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a318f5231747a4a139594fdb65d288, 89e3ac5da4f240a39b32ffa6673277c2; no voters:
I20260928 21:49:57.219413 11140 raft_consensus.cc:2781] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:57.219444 11140 raft_consensus.cc:482] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:57.219455 11140 raft_consensus.cc:3037] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:57.220064 11140 raft_consensus.cc:504] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.220180 11140 leader_election.cc:288] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [CANDIDATE]: Term 1 election: Requested vote from peers 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195), 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949)
I20260928 21:49:57.220329 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1895c982ede04bceadf9c354170b263f" candidate_uuid: "89e3ac5da4f240a39b32ffa6673277c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a318f5231747a4a139594fdb65d288"
I20260928 21:49:57.220394 11045 raft_consensus.cc:3037] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:57.220357 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1895c982ede04bceadf9c354170b263f" candidate_uuid: "89e3ac5da4f240a39b32ffa6673277c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94067b03f8fd417cbd4b26ed3b235313"
I20260928 21:49:57.220432 10778 raft_consensus.cc:3037] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:57.221004 11045 raft_consensus.cc:2454] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e3ac5da4f240a39b32ffa6673277c2 in term 1.
I20260928 21:49:57.221060 10778 raft_consensus.cc:2454] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e3ac5da4f240a39b32ffa6673277c2 in term 1.
I20260928 21:49:57.221179 10850 leader_election.cc:302] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a318f5231747a4a139594fdb65d288, 89e3ac5da4f240a39b32ffa6673277c2; no voters:
I20260928 21:49:57.221293 11140 raft_consensus.cc:2781] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:57.221422 11140 raft_consensus.cc:686] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 LEADER]: Becoming Leader. State: Replica: 89e3ac5da4f240a39b32ffa6673277c2, State: Running, Role: LEADER
I20260928 21:49:57.221496 11140 consensus_queue.cc:237] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.221796 11175 tablet_replica.cc:439] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: TxnStatusTablet state changed. Reason: New leader 89e3ac5da4f240a39b32ffa6673277c2. Latest consensus state: current_term: 1 leader_uuid: "89e3ac5da4f240a39b32ffa6673277c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } }
I20260928 21:49:57.221926 11175 tablet_replica.cc:442] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:49:57.222102 11177 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:49:57.222208 11177 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:49:57.222208 10619 catalog_manager.cc:5495] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194). New cstate: current_term: 1 leader_uuid: "89e3ac5da4f240a39b32ffa6673277c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } health_report { overall_health: UNKNOWN } } }
I20260928 21:49:57.263118 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.260296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:38515 (local address 127.0.0.1:38450)
0928 21:49:57.260374 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:57.260387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:57.260436 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:57.260664 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:57.260665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:57.260666 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:57.260715 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.260717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.261643 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.261644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.262229 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.262232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.262677 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.262677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.262730 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.262745 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.262998 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.263001 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:49:57.263023 (+ 22us) client_negotiation.cc:240] Negotiation successful
0928 21:49:57.263057 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 21:49:57.263131 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.260342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.3.0.194:38515)
0928 21:49:57.260390 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.260392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.260428 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:49:57.260558 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.260560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.260581 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.260657 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:57.260767 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.261514 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.262308 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.262612 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.262682 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.262822 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.262943 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.263006 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.263035 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.263053 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:49:57.264225 10778 raft_consensus.cc:1261] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Refusing update from remote peer 89e3ac5da4f240a39b32ffa6673277c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:57.264274 11045 raft_consensus.cc:1261] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Refusing update from remote peer 89e3ac5da4f240a39b32ffa6673277c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:57.264542 11175 consensus_queue.cc:1035] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:57.264655 11140 consensus_queue.cc:1035] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:57.265110 11139 tablet_replica.cc:439] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: TxnStatusTablet state changed. Reason: New leader 89e3ac5da4f240a39b32ffa6673277c2. Latest consensus state: current_term: 1 leader_uuid: "89e3ac5da4f240a39b32ffa6673277c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } }
I20260928 21:49:57.265201 11139 tablet_replica.cc:442] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:57.265278 11152 tablet_replica.cc:439] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: TxnStatusTablet state changed. Reason: New leader 89e3ac5da4f240a39b32ffa6673277c2. Latest consensus state: current_term: 1 leader_uuid: "89e3ac5da4f240a39b32ffa6673277c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } }
I20260928 21:49:57.265355 11152 tablet_replica.cc:442] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:57.265964 11140 tablet_replica.cc:439] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89e3ac5da4f240a39b32ffa6673277c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } }
I20260928 21:49:57.266043 11140 tablet_replica.cc:442] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:49:57.266335 11139 tablet_replica.cc:439] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89e3ac5da4f240a39b32ffa6673277c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } }
I20260928 21:49:57.266410 11139 tablet_replica.cc:442] T 1895c982ede04bceadf9c354170b263f P 94067b03f8fd417cbd4b26ed3b235313: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:57.267482 11175 tablet_replica.cc:439] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89e3ac5da4f240a39b32ffa6673277c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } }
I20260928 21:49:57.267560 11175 tablet_replica.cc:442] T 1895c982ede04bceadf9c354170b263f P 89e3ac5da4f240a39b32ffa6673277c2: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:49:57.267885 11152 tablet_replica.cc:439] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89e3ac5da4f240a39b32ffa6673277c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } } }
I20260928 21:49:57.267969 11152 tablet_replica.cc:442] T 1895c982ede04bceadf9c354170b263f P 77a318f5231747a4a139594fdb65d288: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:49:57.270172 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.267372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.3.0.193:33195)
0928 21:49:57.267475 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.267477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.267484 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:49:57.267518 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.267518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.267537 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.267594 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:57.268091 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.268629 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.269466 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.269665 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.269726 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.269859 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.269954 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.270014 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.270052 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.270075 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 21:49:57.275022 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.272519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:38515 (local address 127.3.0.193:54569)
0928 21:49:57.272660 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:57.272670 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:57.272727 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:57.272838 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:57.272839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:57.272840 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:57.272885 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.272886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.273692 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.273693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.274273 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.274274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.274715 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.274715 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.274758 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.274770 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.274902 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.274903 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:49:57.274931 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 21:49:57.274960 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260928 21:49:57.275027 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.272558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:54569 (local address 127.3.0.194:38515)
0928 21:49:57.272618 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.272620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.272727 (+ 107us) server_negotiation.cc:407] Connection header received
0928 21:49:57.272765 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.272766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.272775 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.272837 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:57.272919 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.273629 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.274340 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.274642 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.274697 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.274827 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.274856 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.274894 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.274933 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.274950 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:49:57.277225 11045 raft_consensus.cc:1261] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Refusing update from remote peer 94067b03f8fd417cbd4b26ed3b235313: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:57.277252 10918 raft_consensus.cc:1261] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Refusing update from remote peer 94067b03f8fd417cbd4b26ed3b235313: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:57.277468 11162 consensus_queue.cc:1035] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:57.277588 11139 consensus_queue.cc:1035] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:57.279458 11146 mvcc.cc:205] Tried to move back new op lower bound from 7334429480045813760 to 7334429478836056064. Current Snapshot: MvccSnapshot[applied={T|T < 7334429480045813760}]
I20260928 21:49:57.286203 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.283654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:42403 (local address 127.3.0.193:33195)
0928 21:49:57.283704 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.283706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.283775 (+ 69us) server_negotiation.cc:407] Connection header received
0928 21:49:57.283846 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.283847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.283854 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.283893 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:57.284004 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.284439 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.285577 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.285760 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.285805 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.285970 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.286013 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.286051 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.286113 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.286134 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:49:57.286224 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.283484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:33195 (local address 127.3.0.194:42403)
0928 21:49:57.283717 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:57.283729 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:57.283791 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:57.283900 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:57.283900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:57.283902 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:57.283948 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.283949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.284524 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.284525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.285503 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.285505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.285824 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.285825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.285882 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.285896 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.286055 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.286057 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:49:57.286106 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:49:57.286149 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260928 21:49:57.295152 10619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:57.295267 10619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:57.295905 10619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b: Sending DeleteTablet for 3 replicas of tablet 91aea9af814544a2b80a2f9930491834
I20260928 21:49:57.296156 10758 tablet_service.cc:1514] Processing DeleteTablet for tablet 91aea9af814544a2b80a2f9930491834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:57 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:50660
I20260928 21:49:57.296305 11155 tablet_replica.cc:330] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: stopping tablet replica
I20260928 21:49:57.296375 11155 raft_consensus.cc:2229] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:57.296481 11155 raft_consensus.cc:2258] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:57.296615 11027 tablet_service.cc:1514] Processing DeleteTablet for tablet 91aea9af814544a2b80a2f9930491834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:57 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:34734
I20260928 21:49:57.296707 11157 tablet_replica.cc:330] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288: stopping tablet replica
I20260928 21:49:57.296763 11157 raft_consensus.cc:2229] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:57.296792 11157 raft_consensus.cc:2258] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:57.296849 11155 ts_tablet_manager.cc:1905] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:57.297055 11157 ts_tablet_manager.cc:1905] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:57.298512 11155 ts_tablet_manager.cc:1918] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:57.298573 11155 log.cc:1196] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/91aea9af814544a2b80a2f9930491834
I20260928 21:49:57.298782 11155 ts_tablet_manager.cc:1939] T 91aea9af814544a2b80a2f9930491834 P 94067b03f8fd417cbd4b26ed3b235313: Deleting consensus metadata
I20260928 21:49:57.300800 10898 tablet_service.cc:1514] Processing DeleteTablet for tablet 91aea9af814544a2b80a2f9930491834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:57 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:38442
I20260928 21:49:57.300894 11156 tablet_replica.cc:330] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2: stopping tablet replica
I20260928 21:49:57.300948 11156 raft_consensus.cc:2229] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:57.300982 11156 raft_consensus.cc:2258] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:57.300977 11157 ts_tablet_manager.cc:1918] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:57.301024 11157 log.cc:1196] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/91aea9af814544a2b80a2f9930491834
I20260928 21:49:57.301214 11157 ts_tablet_manager.cc:1939] T 91aea9af814544a2b80a2f9930491834 P 77a318f5231747a4a139594fdb65d288: Deleting consensus metadata
I20260928 21:49:57.302335 10607 catalog_manager.cc:4841] TS 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195): tablet 91aea9af814544a2b80a2f9930491834 (table test-table [id=f910d14479c2424fa016104f4f0cc8f5]) successfully deleted
I20260928 21:49:57.302588 10606 catalog_manager.cc:4841] TS 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949): tablet 91aea9af814544a2b80a2f9930491834 (table test-table [id=f910d14479c2424fa016104f4f0cc8f5]) successfully deleted
I20260928 21:49:57.302769 11156 ts_tablet_manager.cc:1905] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:57.304288 11156 ts_tablet_manager.cc:1918] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:57.304335 11156 log.cc:1196] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/91aea9af814544a2b80a2f9930491834
I20260928 21:49:57.304549 11156 ts_tablet_manager.cc:1939] T 91aea9af814544a2b80a2f9930491834 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting consensus metadata
I20260928 21:49:57.306071 10606 catalog_manager.cc:4841] TS 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515): tablet 91aea9af814544a2b80a2f9930491834 (table test-table [id=f910d14479c2424fa016104f4f0cc8f5]) successfully deleted
Sep 28 21:49:57 dist-test-slave-jrck krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:49:57 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:49:57 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:49:57.331984 10671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:49:57.325159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.3.0.254:33381)
0928 21:49:57.325208 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.325210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.325374 (+ 164us) server_negotiation.cc:407] Connection header received
0928 21:49:57.325436 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.325437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.325450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.325513 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:57.326219 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.327018 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.328125 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.328428 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.328585 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.331920 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60922: BlockingRecv error: recv got EOF from 127.0.0.1:60922 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 21:49:57 dist-test-slave-jrck krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM
I20260928 21:49:57.341802 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.333295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.3.0.254:33381)
0928 21:49:57.333342 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.333344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.333456 (+ 112us) server_negotiation.cc:407] Connection header received
0928 21:49:57.333507 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.333508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.333518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.333578 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:49:57.334155 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.334850 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.335863 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.336125 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.336198 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.338881 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:49:57.338883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:49:57.338884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:49:57.338893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:49:57.340606 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:57.341015 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:57.341016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:57.341017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:57.341082 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:49:57.341273 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:49:57.341274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:49:57.341274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:49:57.341408 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:49:57.341504 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.341628 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.341708 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:49:57.343360 10619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:57.343551 10619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:57.345103 10758 tablet_service.cc:1467] Processing CreateTablet for tablet 909294859f044ff48d2592b91e252cb0 (DEFAULT_TABLE table=test-table [id=6e254c71736e426db269ace81f0cb7cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:57.345250 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909294859f044ff48d2592b91e252cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:57.345229 10898 tablet_service.cc:1467] Processing CreateTablet for tablet 909294859f044ff48d2592b91e252cb0 (DEFAULT_TABLE table=test-table [id=6e254c71736e426db269ace81f0cb7cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:57.345356 10898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909294859f044ff48d2592b91e252cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:57.346623 11135 tablet_bootstrap.cc:492] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313: Bootstrap starting.
I20260928 21:49:57.346664 11027 tablet_service.cc:1467] Processing CreateTablet for tablet 909294859f044ff48d2592b91e252cb0 (DEFAULT_TABLE table=test-table [id=6e254c71736e426db269ace81f0cb7cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:57.346783 11027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909294859f044ff48d2592b91e252cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:57.347082 11135 tablet_bootstrap.cc:654] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:57.347724 11135 tablet_bootstrap.cc:492] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313: No bootstrap required, opened a new log
I20260928 21:49:57.347766 11135 ts_tablet_manager.cc:1397] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:57.347911 11135 raft_consensus.cc:348] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.347978 11135 raft_consensus.cc:374] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:57.347997 11135 raft_consensus.cc:729] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94067b03f8fd417cbd4b26ed3b235313, State: Initialized, Role: FOLLOWER
I20260928 21:49:57.348008 11132 tablet_bootstrap.cc:492] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2: Bootstrap starting.
I20260928 21:49:57.348039 11135 consensus_queue.cc:260] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.348119 11135 ts_tablet_manager.cc:1428] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 21:49:57.348392 11132 tablet_bootstrap.cc:654] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:57.348639 11133 tablet_bootstrap.cc:492] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288: Bootstrap starting.
I20260928 21:49:57.348986 11133 tablet_bootstrap.cc:654] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:57.348990 11132 tablet_bootstrap.cc:492] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2: No bootstrap required, opened a new log
I20260928 21:49:57.349032 11132 ts_tablet_manager.cc:1397] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:57.349165 11132 raft_consensus.cc:348] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.349220 11132 raft_consensus.cc:374] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:57.349234 11132 raft_consensus.cc:729] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e3ac5da4f240a39b32ffa6673277c2, State: Initialized, Role: FOLLOWER
I20260928 21:49:57.349269 11132 consensus_queue.cc:260] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.349334 11132 ts_tablet_manager.cc:1428] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:57.350752 11133 tablet_bootstrap.cc:492] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288: No bootstrap required, opened a new log
I20260928 21:49:57.350806 11133 ts_tablet_manager.cc:1397] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:49:57.350915 11133 raft_consensus.cc:348] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.350981 11133 raft_consensus.cc:374] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:57.351001 11133 raft_consensus.cc:729] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a318f5231747a4a139594fdb65d288, State: Initialized, Role: FOLLOWER
I20260928 21:49:57.351042 11133 consensus_queue.cc:260] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.351123 11133 ts_tablet_manager.cc:1428] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 21:49:57.447860 11152 raft_consensus.cc:482] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:57.447956 11152 raft_consensus.cc:504] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.448096 11152 leader_election.cc:288] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195), 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515)
I20260928 21:49:57.448328 10918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909294859f044ff48d2592b91e252cb0" candidate_uuid: "77a318f5231747a4a139594fdb65d288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e3ac5da4f240a39b32ffa6673277c2" is_pre_election: true
I20260928 21:49:57.448340 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909294859f044ff48d2592b91e252cb0" candidate_uuid: "77a318f5231747a4a139594fdb65d288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94067b03f8fd417cbd4b26ed3b235313" is_pre_election: true
I20260928 21:49:57.448423 10918 raft_consensus.cc:2454] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a318f5231747a4a139594fdb65d288 in term 0.
I20260928 21:49:57.448426 10778 raft_consensus.cc:2454] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a318f5231747a4a139594fdb65d288 in term 0.
I20260928 21:49:57.448587 10990 leader_election.cc:302] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a318f5231747a4a139594fdb65d288, 89e3ac5da4f240a39b32ffa6673277c2; no voters:
I20260928 21:49:57.448675 11152 raft_consensus.cc:2781] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:57.448720 11152 raft_consensus.cc:482] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:57.448740 11152 raft_consensus.cc:3037] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:57.449420 11152 raft_consensus.cc:504] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.449539 11152 leader_election.cc:288] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [CANDIDATE]: Term 1 election: Requested vote from peers 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195), 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515)
I20260928 21:49:57.449715 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909294859f044ff48d2592b91e252cb0" candidate_uuid: "77a318f5231747a4a139594fdb65d288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94067b03f8fd417cbd4b26ed3b235313"
I20260928 21:49:57.449723 10918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909294859f044ff48d2592b91e252cb0" candidate_uuid: "77a318f5231747a4a139594fdb65d288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89e3ac5da4f240a39b32ffa6673277c2"
I20260928 21:49:57.449782 10778 raft_consensus.cc:3037] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:57.449788 10918 raft_consensus.cc:3037] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:57.450325 10918 raft_consensus.cc:2454] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a318f5231747a4a139594fdb65d288 in term 1.
I20260928 21:49:57.450338 10778 raft_consensus.cc:2454] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a318f5231747a4a139594fdb65d288 in term 1.
I20260928 21:49:57.450474 10990 leader_election.cc:302] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 77a318f5231747a4a139594fdb65d288, 89e3ac5da4f240a39b32ffa6673277c2, 94067b03f8fd417cbd4b26ed3b235313; no voters:
I20260928 21:49:57.450578 11152 raft_consensus.cc:2781] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:57.450764 11152 raft_consensus.cc:686] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 1 LEADER]: Becoming Leader. State: Replica: 77a318f5231747a4a139594fdb65d288, State: Running, Role: LEADER
I20260928 21:49:57.450847 11152 consensus_queue.cc:237] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.451371 10620 catalog_manager.cc:5495] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77a318f5231747a4a139594fdb65d288 (127.3.0.195). New cstate: current_term: 1 leader_uuid: "77a318f5231747a4a139594fdb65d288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } health_report { overall_health: HEALTHY } } }
I20260928 21:49:57.465441 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.463385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.3.0.195:35949)
0928 21:49:57.463439 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.463441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.463450 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:49:57.463486 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.463487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.463495 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.463531 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:57.463625 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.464098 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.464913 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.465114 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.465175 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.465272 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.465297 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.465328 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.465364 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.465381 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:49:57.466138 10918 raft_consensus.cc:1261] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Refusing update from remote peer 77a318f5231747a4a139594fdb65d288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:57.466150 10778 raft_consensus.cc:1261] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Refusing update from remote peer 77a318f5231747a4a139594fdb65d288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:57.466425 11208 consensus_queue.cc:1035] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:57.466531 11152 consensus_queue.cc:1035] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:57.469014 10619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:57.469105 10619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:57.469624 10619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b: Sending DeleteTablet for 3 replicas of tablet 909294859f044ff48d2592b91e252cb0
I20260928 21:49:57.469816 10758 tablet_service.cc:1514] Processing DeleteTablet for tablet 909294859f044ff48d2592b91e252cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:57 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:50660
I20260928 21:49:57.469867 10898 tablet_service.cc:1514] Processing DeleteTablet for tablet 909294859f044ff48d2592b91e252cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:57 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:38442
I20260928 21:49:57.469862 11027 tablet_service.cc:1514] Processing DeleteTablet for tablet 909294859f044ff48d2592b91e252cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:57 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:34734
I20260928 21:49:57.469899 11155 tablet_replica.cc:330] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313: stopping tablet replica
I20260928 21:49:57.469934 11156 tablet_replica.cc:330] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2: stopping tablet replica
I20260928 21:49:57.469945 11155 raft_consensus.cc:2229] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:57.469944 11157 tablet_replica.cc:330] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288: stopping tablet replica
I20260928 21:49:57.469969 11156 raft_consensus.cc:2229] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:57.469980 11155 raft_consensus.cc:2258] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:57.469991 11157 raft_consensus.cc:2229] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:57.470011 11156 raft_consensus.cc:2258] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:57.470053 11157 raft_consensus.cc:2258] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:57.470038 10619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:49:57.470183 10619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:49:57.470301 11156 ts_tablet_manager.cc:1905] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:57.470474 11157 ts_tablet_manager.cc:1905] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:57.470582 11155 ts_tablet_manager.cc:1905] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:57.471225 11027 tablet_service.cc:1467] Processing CreateTablet for tablet d6bba1360c3e4e58a2c5239d1be77695 (DEFAULT_TABLE table=test-table [id=c9e052457a714e3f9be60c95c12f36fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:57.471321 11027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6bba1360c3e4e58a2c5239d1be77695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:57.471728 11156 ts_tablet_manager.cc:1918] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:57.471791 11156 log.cc:1196] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/909294859f044ff48d2592b91e252cb0
I20260928 21:49:57.471978 11157 ts_tablet_manager.cc:1918] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:57.471987 11156 ts_tablet_manager.cc:1939] T 909294859f044ff48d2592b91e252cb0 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting consensus metadata
I20260928 21:49:57.471987 11155 ts_tablet_manager.cc:1918] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:49:57.472021 11157 log.cc:1196] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/909294859f044ff48d2592b91e252cb0
I20260928 21:49:57.472029 11155 log.cc:1196] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/909294859f044ff48d2592b91e252cb0
I20260928 21:49:57.472218 11157 ts_tablet_manager.cc:1939] T 909294859f044ff48d2592b91e252cb0 P 77a318f5231747a4a139594fdb65d288: Deleting consensus metadata
I20260928 21:49:57.472229 11155 ts_tablet_manager.cc:1939] T 909294859f044ff48d2592b91e252cb0 P 94067b03f8fd417cbd4b26ed3b235313: Deleting consensus metadata
I20260928 21:49:57.472426 11133 tablet_bootstrap.cc:492] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288: Bootstrap starting.
I20260928 21:49:57.472599 10898 tablet_service.cc:1467] Processing CreateTablet for tablet d6bba1360c3e4e58a2c5239d1be77695 (DEFAULT_TABLE table=test-table [id=c9e052457a714e3f9be60c95c12f36fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:57.472690 10898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6bba1360c3e4e58a2c5239d1be77695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:57.472756 11133 tablet_bootstrap.cc:654] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:57.472824 10758 tablet_service.cc:1467] Processing CreateTablet for tablet d6bba1360c3e4e58a2c5239d1be77695 (DEFAULT_TABLE table=test-table [id=c9e052457a714e3f9be60c95c12f36fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:49:57.472908 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6bba1360c3e4e58a2c5239d1be77695. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:49:57.473311 11133 tablet_bootstrap.cc:492] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288: No bootstrap required, opened a new log
I20260928 21:49:57.473351 11133 ts_tablet_manager.cc:1397] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:57.473440 11133 raft_consensus.cc:348] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.473495 11133 raft_consensus.cc:374] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:57.473526 11133 raft_consensus.cc:729] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a318f5231747a4a139594fdb65d288, State: Initialized, Role: FOLLOWER
I20260928 21:49:57.473568 11133 consensus_queue.cc:260] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.473651 11133 ts_tablet_manager.cc:1428] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:57.473765 11132 tablet_bootstrap.cc:492] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2: Bootstrap starting.
I20260928 21:49:57.473796 11135 tablet_bootstrap.cc:492] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313: Bootstrap starting.
I20260928 21:49:57.473877 10607 catalog_manager.cc:4841] TS 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195): tablet 909294859f044ff48d2592b91e252cb0 (table test-table [id=6e254c71736e426db269ace81f0cb7cb]) successfully deleted
I20260928 21:49:57.474061 11135 tablet_bootstrap.cc:654] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:57.474061 11132 tablet_bootstrap.cc:654] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2: Neither blocks nor log segments found. Creating new log.
I20260928 21:49:57.474352 10606 catalog_manager.cc:4841] TS 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949): tablet 909294859f044ff48d2592b91e252cb0 (table test-table [id=6e254c71736e426db269ace81f0cb7cb]) successfully deleted
I20260928 21:49:57.474412 10606 catalog_manager.cc:4841] TS 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515): tablet 909294859f044ff48d2592b91e252cb0 (table test-table [id=6e254c71736e426db269ace81f0cb7cb]) successfully deleted
I20260928 21:49:57.474577 11135 tablet_bootstrap.cc:492] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313: No bootstrap required, opened a new log
I20260928 21:49:57.474627 11135 ts_tablet_manager.cc:1397] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:57.474741 11135 raft_consensus.cc:348] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.474797 11135 raft_consensus.cc:374] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:57.474810 11135 raft_consensus.cc:729] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94067b03f8fd417cbd4b26ed3b235313, State: Initialized, Role: FOLLOWER
I20260928 21:49:57.474856 11135 consensus_queue.cc:260] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.474936 11135 ts_tablet_manager.cc:1428] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:57.475117 11132 tablet_bootstrap.cc:492] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2: No bootstrap required, opened a new log
I20260928 21:49:57.475162 11132 ts_tablet_manager.cc:1397] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:49:57.475286 11132 raft_consensus.cc:348] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.475353 11132 raft_consensus.cc:374] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:49:57.475373 11132 raft_consensus.cc:729] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e3ac5da4f240a39b32ffa6673277c2, State: Initialized, Role: FOLLOWER
I20260928 21:49:57.475415 11132 consensus_queue.cc:260] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.475498 11132 ts_tablet_manager.cc:1428] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:49:57.710345 11140 raft_consensus.cc:482] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:49:57.710434 11140 raft_consensus.cc:504] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.710573 11140 leader_election.cc:288] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195), 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949)
I20260928 21:49:57.710795 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6bba1360c3e4e58a2c5239d1be77695" candidate_uuid: "89e3ac5da4f240a39b32ffa6673277c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94067b03f8fd417cbd4b26ed3b235313" is_pre_election: true
I20260928 21:49:57.710816 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6bba1360c3e4e58a2c5239d1be77695" candidate_uuid: "89e3ac5da4f240a39b32ffa6673277c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a318f5231747a4a139594fdb65d288" is_pre_election: true
I20260928 21:49:57.710891 11045 raft_consensus.cc:2454] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e3ac5da4f240a39b32ffa6673277c2 in term 0.
I20260928 21:49:57.710896 10778 raft_consensus.cc:2454] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89e3ac5da4f240a39b32ffa6673277c2 in term 0.
I20260928 21:49:57.711045 10850 leader_election.cc:302] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a318f5231747a4a139594fdb65d288, 89e3ac5da4f240a39b32ffa6673277c2; no voters:
I20260928 21:49:57.711165 11140 raft_consensus.cc:2781] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:49:57.711213 11140 raft_consensus.cc:482] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:49:57.711227 11140 raft_consensus.cc:3037] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:57.711884 11140 raft_consensus.cc:504] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.711993 11140 leader_election.cc:288] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [CANDIDATE]: Term 1 election: Requested vote from peers 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195), 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949)
I20260928 21:49:57.712136 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6bba1360c3e4e58a2c5239d1be77695" candidate_uuid: "89e3ac5da4f240a39b32ffa6673277c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a318f5231747a4a139594fdb65d288"
I20260928 21:49:57.712148 10778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6bba1360c3e4e58a2c5239d1be77695" candidate_uuid: "89e3ac5da4f240a39b32ffa6673277c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94067b03f8fd417cbd4b26ed3b235313"
I20260928 21:49:57.712203 11045 raft_consensus.cc:3037] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:57.712216 10778 raft_consensus.cc:3037] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:49:57.712807 11045 raft_consensus.cc:2454] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e3ac5da4f240a39b32ffa6673277c2 in term 1.
I20260928 21:49:57.712865 10778 raft_consensus.cc:2454] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89e3ac5da4f240a39b32ffa6673277c2 in term 1.
I20260928 21:49:57.712985 10850 leader_election.cc:302] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a318f5231747a4a139594fdb65d288, 89e3ac5da4f240a39b32ffa6673277c2; no voters:
I20260928 21:49:57.713078 11140 raft_consensus.cc:2781] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:49:57.713127 11140 raft_consensus.cc:686] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 LEADER]: Becoming Leader. State: Replica: 89e3ac5da4f240a39b32ffa6673277c2, State: Running, Role: LEADER
I20260928 21:49:57.713176 11140 consensus_queue.cc:237] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } }
I20260928 21:49:57.713723 10620 catalog_manager.cc:5495] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194). New cstate: current_term: 1 leader_uuid: "89e3ac5da4f240a39b32ffa6673277c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89e3ac5da4f240a39b32ffa6673277c2" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 } health_report { overall_health: UNKNOWN } } }
I20260928 21:49:57.760633 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.758769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.3.0.194:38515)
0928 21:49:57.758807 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.758809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.758817 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:49:57.758866 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.758867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.758874 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.758927 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:57.759039 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.759486 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.760128 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.760296 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.760338 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.760442 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.760481 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.760521 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.760559 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.760576 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:49:57.764228 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.762370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:43867 (local address 127.3.0.194:38515)
0928 21:49:57.762402 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 21:49:57.762404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:49:57.762438 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:49:57.762535 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:49:57.762536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:49:57.762544 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:49:57.762590 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:49:57.762696 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.763114 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.763762 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.763928 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.763976 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.764059 (+ 83us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.764080 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.764109 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 21:49:57.764140 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 21:49:57.764151 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:49:57.764235 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.762338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:38515 (local address 127.3.0.194:43867)
0928 21:49:57.762398 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:57.762409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:49:57.762451 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:49:57.762599 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:49:57.762600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:49:57.762601 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:49:57.762640 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.762641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.763163 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.763163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.763689 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:49:57.763691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:49:57.763981 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:49:57.763981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:49:57.764016 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:49:57.764021 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:49:57.764109 (+ 88us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:49:57.764110 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 21:49:57.764133 (+ 23us) client_negotiation.cc:240] Negotiation successful
0928 21:49:57.764163 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 21:49:57.765951 10778 raft_consensus.cc:1261] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Refusing update from remote peer 89e3ac5da4f240a39b32ffa6673277c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:57.765995 11045 raft_consensus.cc:1261] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Refusing update from remote peer 89e3ac5da4f240a39b32ffa6673277c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:49:57.766119 11175 consensus_queue.cc:1035] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94067b03f8fd417cbd4b26ed3b235313" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:57.766247 11140 consensus_queue.cc:1035] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a318f5231747a4a139594fdb65d288" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 35949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:49:57.776077 10619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:57.776176 10619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:49:57.776765 10619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1286be4222441558a1bfb2145802e5b: Sending DeleteTablet for 3 replicas of tablet d6bba1360c3e4e58a2c5239d1be77695
I20260928 21:49:57.776993 10898 tablet_service.cc:1514] Processing DeleteTablet for tablet d6bba1360c3e4e58a2c5239d1be77695 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:57 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:38442
I20260928 21:49:57.776993 11027 tablet_service.cc:1514] Processing DeleteTablet for tablet d6bba1360c3e4e58a2c5239d1be77695 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:57 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:34734
I20260928 21:49:57.777093 11157 tablet_replica.cc:330] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288: stopping tablet replica
I20260928 21:49:57.777155 11157 raft_consensus.cc:2229] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:57.777194 11157 raft_consensus.cc:2258] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:57.777261 11156 tablet_replica.cc:330] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2: stopping tablet replica
I20260928 21:49:57.777311 11156 raft_consensus.cc:2229] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:49:57.777391 11156 raft_consensus.cc:2258] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:57.777695 11157 ts_tablet_manager.cc:1905] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:57.779217 11157 ts_tablet_manager.cc:1918] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:57.779237 11156 ts_tablet_manager.cc:1905] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:57.779266 11157 log.cc:1196] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/d6bba1360c3e4e58a2c5239d1be77695
I20260928 21:49:57.779448 11157 ts_tablet_manager.cc:1939] T d6bba1360c3e4e58a2c5239d1be77695 P 77a318f5231747a4a139594fdb65d288: Deleting consensus metadata
I20260928 21:49:57.779762 10606 catalog_manager.cc:4841] TS 77a318f5231747a4a139594fdb65d288 (127.3.0.195:35949): tablet d6bba1360c3e4e58a2c5239d1be77695 (table test-table [id=c9e052457a714e3f9be60c95c12f36fd]) successfully deleted
I20260928 21:49:57.780673 11156 ts_tablet_manager.cc:1918] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:57.780737 11156 log.cc:1196] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/d6bba1360c3e4e58a2c5239d1be77695
I20260928 21:49:57.780926 11156 ts_tablet_manager.cc:1939] T d6bba1360c3e4e58a2c5239d1be77695 P 89e3ac5da4f240a39b32ffa6673277c2: Deleting consensus metadata
I20260928 21:49:57.781261 10758 tablet_service.cc:1514] Processing DeleteTablet for tablet d6bba1360c3e4e58a2c5239d1be77695 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:49:57 UTC) from {username='oryx', principal='oryx/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:50660
I20260928 21:49:57.781250 10606 catalog_manager.cc:4841] TS 89e3ac5da4f240a39b32ffa6673277c2 (127.3.0.194:38515): tablet d6bba1360c3e4e58a2c5239d1be77695 (table test-table [id=c9e052457a714e3f9be60c95c12f36fd]) successfully deleted
I20260928 21:49:57.781348 11155 tablet_replica.cc:330] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313: stopping tablet replica
I20260928 21:49:57.781388 11155 raft_consensus.cc:2229] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:49:57.781421 11155 raft_consensus.cc:2258] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:49:57.781857 11155 ts_tablet_manager.cc:1905] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:49:57.783232 11155 ts_tablet_manager.cc:1918] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:49:57.783284 11155 log.cc:1196] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/d6bba1360c3e4e58a2c5239d1be77695
I20260928 21:49:57.783360 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 10693
I20260928 21:49:57.783464 11155 ts_tablet_manager.cc:1939] T d6bba1360c3e4e58a2c5239d1be77695 P 94067b03f8fd417cbd4b26ed3b235313: Deleting consensus metadata
W20260928 21:49:57.791915 10607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:33195 (error 108)
I20260928 21:49:57.792163 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 10833
I20260928 21:49:57.792985 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:49:57.792813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:33195 (local address 127.0.0.1:50684)
0928 21:49:57.792901 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 21:49:57.792952 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":36,"threads_started":1}
W20260928 21:49:57.793124 10607 catalog_manager.cc:4584] TS 94067b03f8fd417cbd4b26ed3b235313 (127.3.0.193:33195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6bba1360c3e4e58a2c5239d1be77695: Network error: Client connection negotiation failed: client connection to 127.3.0.193:33195: connect: Connection refused (error 111)
I20260928 21:49:57.798053 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 10973
I20260928 21:49:57.802620 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 10589
2026-09-28T21:49:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2686 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:57 dist-test-slave-jrck krb5kdc[11227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:57 dist-test-slave-jrck krb5kdc[11227](info): set up 2 sockets
Sep 28 21:49:57 dist-test-slave-jrck krb5kdc[11227](info): commencing operation
krb5kdc: starting...
W20260928 21:49:58.752556 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:58 dist-test-slave-jrck krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:58.858413 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:42123
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:42123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:49:58.935451 11240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:49:58.935653 11240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:49:58.935673 11240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:49:58.936792 11240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:49:58.936825 11240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:49:58.936836 11240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:49:58.936847 11240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:49:58.936857 11240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:49:58.938287 11240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:42123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:49:58.938472 11240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:49:58.938627 11240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:49:58.943034 11248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:58.943049 11246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:49:58.943046 11245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:49:58.943189 11240 server_base.cc:1034] running on GCE node
I20260928 21:49:58.943493 11240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:49:58.943740 11240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:49:58.944880 11240 hybrid_clock.cc:648] HybridClock initialized: now 1790632198944866 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T21:49:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1143 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:49:58 dist-test-slave-jrck krb5kdc[11253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:49:58 dist-test-slave-jrck krb5kdc[11253](info): set up 2 sockets
Sep 28 21:49:58 dist-test-slave-jrck krb5kdc[11253](info): commencing operation
krb5kdc: starting...
W20260928 21:49:59.888685 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:49:59 dist-test-slave-jrck krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:49:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:49:59.995256 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38607
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:38607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:50:00.075156 11266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:00.075397 11266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:00.075430 11266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:00.076617 11266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:50:00.076663 11266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:00.076682 11266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:00.076699 11266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:50:00.076721 11266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:50:00.078197 11266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:38607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:00.078442 11266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:00.078639 11266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:00.083133 11272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.083123 11274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.083186 11271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:00.083340 11266 server_base.cc:1034] running on GCE node
I20260928 21:50:00.083575 11266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:00.083842 11266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:00.084971 11266 hybrid_clock.cc:648] HybridClock initialized: now 1790632200084957 us; error 32 us; skew 500 ppm
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632200, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:00.087344 11266 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:50:00.087749 11266 webserver.cc:460] Webserver started at http://127.3.0.254:44583/ using document root <none> and password file <none>
I20260928 21:50:00.087922 11266 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:00.087960 11266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:00.088047 11266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:00.089071 11266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "4378acd899bf65874297a079fce3d3e5"
server_key_iv: "8911690a595668435d6626eabeea0024"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.089367 11266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "4378acd899bf65874297a079fce3d3e5"
server_key_iv: "8911690a595668435d6626eabeea0024"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.090492 11266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:50:00.091074 11281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.091213 11266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:00.091303 11266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "4378acd899bf65874297a079fce3d3e5"
server_key_iv: "8911690a595668435d6626eabeea0024"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.091367 11266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:00.104672 11266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:00.106591 11266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:00.106724 11266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:00.110358 11266 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:38607
I20260928 21:50:00.110392 11333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:38607 every 8 connection(s)
I20260928 21:50:00.110687 11266 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:50:00.111467 11334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:50:00.113662 11334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a: Bootstrap starting.
I20260928 21:50:00.114149 11334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a: Neither blocks nor log segments found. Creating new log.
I20260928 21:50:00.114374 11334 log.cc:824] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:50:00.114936 11334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a: No bootstrap required, opened a new log
I20260928 21:50:00.116428 11334 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } }
I20260928 21:50:00.116607 11334 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:50:00.116637 11334 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b3cac1fb2a4457ab04a2f5493a3a2a, State: Initialized, Role: FOLLOWER
I20260928 21:50:00.116757 11334 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } }
I20260928 21:50:00.116825 11334 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:50:00.116851 11334 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:50:00.116881 11334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:50:00.117318 11334 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } }
I20260928 21:50:00.117398 11334 leader_election.cc:302] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5b3cac1fb2a4457ab04a2f5493a3a2a; no voters:
I20260928 21:50:00.117512 11334 leader_election.cc:288] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:50:00.117560 11337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:50:00.117727 11334 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:50:00.117938 11337 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 1 LEADER]: Becoming Leader. State: Replica: b5b3cac1fb2a4457ab04a2f5493a3a2a, State: Running, Role: LEADER
I20260928 21:50:00.118028 11337 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } }
I20260928 21:50:00.118383 11338 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } } }
I20260928 21:50:00.118389 11339 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5b3cac1fb2a4457ab04a2f5493a3a2a. Latest consensus state: current_term: 1 leader_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } } }
I20260928 21:50:00.118485 11338 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [sys.catalog]: This master's current role is: LEADER
I20260928 21:50:00.118495 11339 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [sys.catalog]: This master's current role is: LEADER
I20260928 21:50:00.118680 11346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:50:00.119194 11346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:50:00.119465 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 11266
I20260928 21:50:00.119567 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:50:00.119776 3075 external_mini_cluster.cc:1428] Setting key 695286f2b3954fad68bd8a53d6c9f9cf
I20260928 21:50:00.121284 11346 catalog_manager.cc:1347] Generated new cluster ID: 414944b2b7484079868ac63839cb9742
I20260928 21:50:00.121325 11346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:50:00.136072 11346 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:50:00.136718 11346 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:50:00.143563 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:00.120188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.3.0.254:38607)
0928 21:50:00.120344 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:00.120353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:00.132669 (+ 12316us) server_negotiation.cc:407] Connection header received
0928 21:50:00.132803 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:00.132811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:00.132880 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:00.133026 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:00.133783 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.135338 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.136481 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.137177 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.137284 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.140029 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:00.140034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:00.140035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:50:00.140047 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:00.141978 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:00.142338 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:00.142341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:00.142342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:00.142470 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:00.142637 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:00.142637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:00.142638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:00.142929 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:00.143071 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:00.143167 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 21:50:00.143321 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":174,"server-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260928 21:50:00.149940 11346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a: Generated new TSK 0
I20260928 21:50:00.150098 11346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:50:00.175104 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38607
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:50:00.253661 11360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:00.253865 11360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:00.253886 11360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:00.254994 11360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:00.255028 11360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:00.255051 11360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:50:00.256403 11360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:00.256595 11360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:00.256772 11360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:00.261198 11366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.261204 11365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.261204 11368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:00.261440 11360 server_base.cc:1034] running on GCE node
I20260928 21:50:00.261626 11360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:00.261832 11360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:00.262964 11360 hybrid_clock.cc:648] HybridClock initialized: now 1790632200262954 us; error 27 us; skew 500 ppm
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632200, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:00.272050 11360 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:50:00.272512 11360 webserver.cc:460] Webserver started at http://127.3.0.193:38357/ using document root <none> and password file <none>
I20260928 21:50:00.272708 11360 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:00.272760 11360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:00.272879 11360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:00.273890 11360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "5285011514834d589b8b9ea57321b21c"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "e781aaf30d1f99a939a9e7501c2bf2d6"
server_key_iv: "afc178e81f4a72cfe14c64c312f34ce3"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.274205 11360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "5285011514834d589b8b9ea57321b21c"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "e781aaf30d1f99a939a9e7501c2bf2d6"
server_key_iv: "afc178e81f4a72cfe14c64c312f34ce3"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.275365 11360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:50:00.275990 11375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.276134 11360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:50:00.276212 11360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "5285011514834d589b8b9ea57321b21c"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "e781aaf30d1f99a939a9e7501c2bf2d6"
server_key_iv: "afc178e81f4a72cfe14c64c312f34ce3"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.276278 11360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:00.298046 11360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:00.299839 11360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:00.299973 11360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:00.300181 11360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:00.300498 11360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:00.300544 11360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.300572 11360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:00.300593 11360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.305912 11360 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:36101
I20260928 21:50:00.305971 11488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:36101 every 8 connection(s)
I20260928 21:50:00.306260 11360 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:50:00.309633 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 11360
I20260928 21:50:00.309718 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:50:00.309914 3075 external_mini_cluster.cc:1428] Setting key cdab80d92735b3831383cd7a3601d8fc
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632200, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:50:00.315172 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:00.306871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:39567 (local address 127.3.0.254:38607)
0928 21:50:00.306916 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:00.306917 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:00.307104 (+ 187us) server_negotiation.cc:407] Connection header received
0928 21:50:00.307578 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:00.307579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:00.307589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:00.307641 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:00.308525 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.309030 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.310008 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.310281 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.310342 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.312545 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:00.312546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:00.312547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:50:00.312555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:00.313779 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:00.314133 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:00.314134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:00.314135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:00.314186 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:00.314498 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:00.314500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:00.314500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:00.314639 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:00.314702 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:00.314976 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 21:50:00.315082 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:50:00.315320 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:00.306839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:39567)
0928 21:50:00.306992 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:00.307033 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:00.307501 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:00.307696 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:00.307699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:00.308234 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:00.308424 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.308429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.309095 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.309095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.309937 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.309940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.310356 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.310357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.310474 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.310898 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:00.310906 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:00.312400 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:00.313850 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:00.313857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:00.313859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:00.314069 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:00.314279 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:00.314280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:00.314280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:00.314428 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:00.314703 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:00.314710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:00.314855 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 21:50:00.314947 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:50:00.315106 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260928 21:50:00.315788 11489 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38607
I20260928 21:50:00.315873 11489 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:00.316048 11489 heartbeater.cc:511] Master 127.3.0.254:38607 requested a full tablet report, sending...
I20260928 21:50:00.316591 11298 ts_manager.cc:194] Registered new tserver with Master: 5285011514834d589b8b9ea57321b21c (127.3.0.193:36101)
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
I20260928 21:50:00.317209 11298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:39567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:50:00.332801 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38607
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:50:00.411273 11494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:00.411470 11494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:00.411492 11494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:00.412616 11494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:00.412647 11494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:00.412671 11494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:50:00.414013 11494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:00.414191 11494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:00.414352 11494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:00.418926 11502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:00.418962 11494 server_base.cc:1034] running on GCE node
W20260928 21:50:00.418919 11499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.418919 11500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:00.419286 11494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:00.419545 11494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:00.420684 11494 hybrid_clock.cc:648] HybridClock initialized: now 1790632200420671 us; error 28 us; skew 500 ppm
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632200, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:00.422873 11494 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:50:00.423310 11494 webserver.cc:460] Webserver started at http://127.3.0.194:35291/ using document root <none> and password file <none>
I20260928 21:50:00.423502 11494 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:00.423547 11494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:00.423663 11494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:00.424685 11494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "24f38b28f1374048a3be403b3ac69c49"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "4a776b327f54b059729385769b47e059"
server_key_iv: "cdf56e8240b83aa296f0fbaf4481ae60"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.424999 11494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "24f38b28f1374048a3be403b3ac69c49"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "4a776b327f54b059729385769b47e059"
server_key_iv: "cdf56e8240b83aa296f0fbaf4481ae60"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.426201 11494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:50:00.426882 11509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.427030 11494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:00.427109 11494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "24f38b28f1374048a3be403b3ac69c49"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "4a776b327f54b059729385769b47e059"
server_key_iv: "cdf56e8240b83aa296f0fbaf4481ae60"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.427165 11494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:00.451333 11494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:00.453110 11494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:00.453256 11494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:00.453474 11494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:00.453758 11494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:00.453790 11494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.453819 11494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:00.453835 11494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.459250 11494 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:42395
I20260928 21:50:00.459288 11622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:42395 every 8 connection(s)
I20260928 21:50:00.459579 11494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632200, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:50:00.467319 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 11494
I20260928 21:50:00.467433 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:50:00.467650 3075 external_mini_cluster.cc:1428] Setting key 605d4118557e9a7358b9af5cb16dca73
I20260928 21:50:00.467803 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:00.460200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:45425 (local address 127.3.0.254:38607)
0928 21:50:00.460261 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:00.460263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:00.460440 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:50:00.460954 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:00.460955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:00.460964 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:00.461018 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:00.461876 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.462367 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.463276 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.463449 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.463491 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.465356 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:00.465358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:00.465359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:50:00.465366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:00.466404 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:00.466761 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:00.466762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:00.466763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:00.466814 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:00.467015 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:00.467016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:00.467016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:00.467138 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:00.467201 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:00.467615 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 21:50:00.467715 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:50:00.467846 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:00.460173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:45425)
0928 21:50:00.460319 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:00.460359 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:00.460870 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:00.461074 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:00.461077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:00.461616 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:00.461810 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.461814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.462428 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.462429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.463216 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.463218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.463499 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.463500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.463589 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.463981 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:00.463990 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:00.465267 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:00.466473 (+ 1206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:00.466479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:00.466481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:00.466690 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:00.466867 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:00.466868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:00.466868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:00.466962 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:00.467207 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:00.467212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:00.467330 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 21:50:00.467438 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:50:00.467599 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260928 21:50:00.468256 11623 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38607
I20260928 21:50:00.468355 11623 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:00.468544 11623 heartbeater.cc:511] Master 127.3.0.254:38607 requested a full tablet report, sending...
I20260928 21:50:00.468899 11298 ts_manager.cc:194] Registered new tserver with Master: 24f38b28f1374048a3be403b3ac69c49 (127.3.0.194:42395)
I20260928 21:50:00.469354 11298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:45425
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:50:00.488794 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38607
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:50:00.566972 11628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:00.567193 11628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:00.567216 11628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:00.568449 11628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:00.568507 11628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:00.568533 11628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:50:00.569905 11628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:00.570113 11628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:00.570317 11628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:00.574853 11634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.574903 11633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.574860 11636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:00.574995 11628 server_base.cc:1034] running on GCE node
I20260928 21:50:00.575268 11628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:00.575497 11628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:00.576627 11628 hybrid_clock.cc:648] HybridClock initialized: now 1790632200576610 us; error 33 us; skew 500 ppm
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632200, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:00.579017 11628 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:50:00.579483 11628 webserver.cc:460] Webserver started at http://127.3.0.195:40801/ using document root <none> and password file <none>
I20260928 21:50:00.579701 11628 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:00.579751 11628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:00.579880 11628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:00.581025 11628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "77ec43f3eb5446cebb44f683ed9b71cd"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "d699e9e806c543f5a7caa54bf4a9ff3a"
server_key_iv: "4ca10a7452a8bfad14e14c264fe06857"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.581382 11628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "77ec43f3eb5446cebb44f683ed9b71cd"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "d699e9e806c543f5a7caa54bf4a9ff3a"
server_key_iv: "4ca10a7452a8bfad14e14c264fe06857"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.582849 11628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:50:00.583585 11643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.583756 11628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:00.583850 11628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "77ec43f3eb5446cebb44f683ed9b71cd"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "d699e9e806c543f5a7caa54bf4a9ff3a"
server_key_iv: "4ca10a7452a8bfad14e14c264fe06857"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.583915 11628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:00.609850 11628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:00.611811 11628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:00.611937 11628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:00.612138 11628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:00.612560 11628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:00.612603 11628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.612635 11628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:00.612659 11628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.618710 11628 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:41107
I20260928 21:50:00.618779 11756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:41107 every 8 connection(s)
I20260928 21:50:00.619231 11628 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:50:00.623414 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 11628
I20260928 21:50:00.623536 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:50:00.623762 3075 external_mini_cluster.cc:1428] Setting key fcb3c3c22cef69df8de08f61de83d510
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632200, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:50:00.629676 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:00.620132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:53057 (local address 127.3.0.254:38607)
0928 21:50:00.620182 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:00.620184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:00.620436 (+ 252us) server_negotiation.cc:407] Connection header received
0928 21:50:00.621126 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:00.621128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:00.621142 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:00.621207 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:00.622559 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.623326 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.624855 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.625058 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.625111 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.627259 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:00.627260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:00.627261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:50:00.627268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:00.628411 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:00.628824 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:00.628825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:00.628825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:00.628878 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:00.629090 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:00.629091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:00.629091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:00.629199 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:00.629253 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:00.629503 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 21:50:00.629588 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:50:00.629897 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:00.620136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:53057)
0928 21:50:00.620313 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:00.620358 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:00.621005 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:00.621339 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:00.621344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:00.622122 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:00.622413 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.622421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.623433 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.623434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.624777 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.624780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.625121 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.625122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.625257 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.625751 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:00.625759 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:00.627171 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:00.628499 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:00.628505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:00.628507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:00.628757 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:00.628931 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:00.628932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:00.628932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:00.629034 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:00.629250 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:00.629256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:00.629393 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 21:50:00.629477 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:50:00.629655 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260928 21:50:00.630303 11757 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38607
I20260928 21:50:00.630391 11757 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:00.630585 11757 heartbeater.cc:511] Master 127.3.0.254:38607 requested a full tablet report, sending...
I20260928 21:50:00.630947 11298 ts_manager.cc:194] Registered new tserver with Master: 77ec43f3eb5446cebb44f683ed9b71cd (127.3.0.195:41107)
I20260928 21:50:00.631292 11298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:53057
I20260928 21:50:00.635352 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab.
Entry for principal oryx/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab.
WARNING: no policy specified for oryx/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab.
Entry for principal oryx/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab.
WARNING: no policy specified for oryx/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab.
Entry for principal oryx/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab.
I20260928 21:50:00.697302 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 11360
I20260928 21:50:00.701669 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 11494
I20260928 21:50:00.705395 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 11628
I20260928 21:50:00.708853 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 11266
I20260928 21:50:00.712886 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38607
--webserver_interface=127.3.0.254
--webserver_port=44583
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:38607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:50:00.792003 11766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:00.792335 11766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:00.792403 11766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:00.793982 11766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:50:00.794034 11766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:00.794054 11766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:00.794068 11766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:50:00.794085 11766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:50:00.795810 11766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:38607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:00.796077 11766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:00.796304 11766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:00.800832 11772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.800849 11771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.800843 11774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:00.801066 11766 server_base.cc:1034] running on GCE node
I20260928 21:50:00.801287 11766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:00.801613 11766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:00.802776 11766 hybrid_clock.cc:648] HybridClock initialized: now 1790632200802762 us; error 29 us; skew 500 ppm
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632200, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:00.805076 11766 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:50:00.805497 11766 webserver.cc:460] Webserver started at http://127.3.0.254:44583/ using document root <none> and password file <none>
I20260928 21:50:00.805709 11766 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:00.805771 11766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:00.806998 11766 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:50:00.807548 11781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.807705 11766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.807786 11766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "4378acd899bf65874297a079fce3d3e5"
server_key_iv: "8911690a595668435d6626eabeea0024"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.808113 11766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:00.836208 11766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:00.838016 11766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:00.838150 11766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:00.841851 11766 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:38607
I20260928 21:50:00.841885 11833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:38607 every 8 connection(s)
I20260928 21:50:00.842227 11766 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:50:00.843587 11834 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:50:00.844157 11834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a: Bootstrap starting.
I20260928 21:50:00.847239 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 11766
I20260928 21:50:00.847494 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:36101
--local_ip_for_outbound_sockets=127.3.0.193
--tserver_master_addrs=127.3.0.254:38607
--webserver_port=38357
--webserver_interface=127.3.0.193
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.0.193
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:50:00.852154 11834 log.cc:824] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:50:00.853928 11834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:50:00.854185 11834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a: Bootstrap complete.
I20260928 21:50:00.856215 11834 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } }
I20260928 21:50:00.856437 11834 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b3cac1fb2a4457ab04a2f5493a3a2a, State: Initialized, Role: FOLLOWER
I20260928 21:50:00.856626 11834 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } }
I20260928 21:50:00.856691 11834 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:50:00.856742 11834 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:50:00.856788 11834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:50:00.857728 11834 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } }
I20260928 21:50:00.857817 11834 leader_election.cc:302] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5b3cac1fb2a4457ab04a2f5493a3a2a; no voters:
I20260928 21:50:00.857980 11834 leader_election.cc:288] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:50:00.858028 11838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:50:00.858146 11838 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [term 2 LEADER]: Becoming Leader. State: Replica: b5b3cac1fb2a4457ab04a2f5493a3a2a, State: Running, Role: LEADER
I20260928 21:50:00.858244 11834 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:50:00.858238 11838 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } }
I20260928 21:50:00.858793 11839 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5b3cac1fb2a4457ab04a2f5493a3a2a. Latest consensus state: current_term: 2 leader_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } } }
I20260928 21:50:00.858870 11839 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [sys.catalog]: This master's current role is: LEADER
I20260928 21:50:00.859131 11840 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b3cac1fb2a4457ab04a2f5493a3a2a" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38607 } } }
I20260928 21:50:00.859185 11840 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a [sys.catalog]: This master's current role is: LEADER
I20260928 21:50:00.859174 11847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:50:00.859660 11847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:50:00.859807 11847 catalog_manager.cc:1259] Loaded cluster ID: 414944b2b7484079868ac63839cb9742
I20260928 21:50:00.859992 11847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:50:00.861159 11847 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:50:00.861336 11847 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5b3cac1fb2a4457ab04a2f5493a3a2a: Loaded TSK: 0
I20260928 21:50:00.861550 11847 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:50:00.927209 11836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:00.927428 11836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:00.927459 11836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:00.928719 11836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:00.928761 11836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:00.928797 11836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:50:00.930279 11836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.193.keytab
--principal=oryx/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:36101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=38357
--tserver_master_addrs=127.3.0.254:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:00.930511 11836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:00.930712 11836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:00.936580 11860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.936643 11861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:00.936682 11863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:00.936872 11836 server_base.cc:1034] running on GCE node
I20260928 21:50:00.937048 11836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:00.937283 11836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:00.938432 11836 hybrid_clock.cc:648] HybridClock initialized: now 1790632200938419 us; error 26 us; skew 500 ppm
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632200, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:00.941033 11836 init.cc:377] Logged in from keytab as oryx/127.3.0.193@KRBTEST.COM (short username oryx)
I20260928 21:50:00.941468 11836 webserver.cc:460] Webserver started at http://127.3.0.193:38357/ using document root <none> and password file <none>
I20260928 21:50:00.941670 11836 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:00.941728 11836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:00.943040 11836 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:50:00.943619 11870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.943783 11836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.943863 11836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "5285011514834d589b8b9ea57321b21c"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "e781aaf30d1f99a939a9e7501c2bf2d6"
server_key_iv: "afc178e81f4a72cfe14c64c312f34ce3"
server_key_version: "encryptionkey@0"
I20260928 21:50:00.944202 11836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:00.961256 11836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:00.963380 11836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:00.963512 11836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:00.963719 11836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:00.963992 11836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:00.964023 11836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.964051 11836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:00.964072 11836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:00.969722 11836 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:36101
I20260928 21:50:00.969779 11983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:36101 every 8 connection(s)
I20260928 21:50:00.970088 11836 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:50:00.971961 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 11836
I20260928 21:50:00.972239 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:42395
--local_ip_for_outbound_sockets=127.3.0.194
--tserver_master_addrs=127.3.0.254:38607
--webserver_port=35291
--webserver_interface=127.3.0.194
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.0.194
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:00.981333 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:00.970794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:41753)
0928 21:50:00.970968 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:00.971012 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:00.971500 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:00.973136 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:00.973140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:00.973884 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:00.974152 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.974158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.975326 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.975327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.976620 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.976623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.977507 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.977508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.977681 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.978336 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:00.978346 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:00.981116 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
W20260928 21:50:00.981557 11984 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:38607 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.0.254:38607: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
W20260928 21:50:00.981815 11987 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:00.970833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:41753 (local address 127.3.0.254:38607)
0928 21:50:00.972634 (+ 1801us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:00.972645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:00.972695 (+ 50us) server_negotiation.cc:407] Connection header received
0928 21:50:00.972848 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:00.972855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:00.972912 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:00.973055 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:00.974287 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.975220 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.976729 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.977400 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.977520 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.981302 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:41753: BlockingRecv error: recv got EOF from 127.3.0.193:41753 (error 108)
Metrics: {"server-negotiator.queue_time_us":1726,"thread_start_us":83,"threads_started":1}
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:00.987054 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:00.981784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:44245)
0928 21:50:00.982132 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:00.982142 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:00.982191 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:00.982327 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:00.982328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:00.982469 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:00.982521 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.982522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.983404 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.983405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.984153 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.984156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.984629 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.984630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.984691 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.985123 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:00.985127 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:00.986983 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331}
W20260928 21:50:00.987198 11987 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:00.981870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:44245 (local address 127.3.0.254:38607)
0928 21:50:00.981913 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:00.981915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:00.982194 (+ 279us) server_negotiation.cc:407] Connection header received
0928 21:50:00.982243 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:00.982244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:00.982257 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:00.982320 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:00.982582 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.983329 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.984239 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.984570 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.984628 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.987079 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:44245: BlockingRecv error: recv got EOF from 127.3.0.193:44245 (error 108)
Metrics: {"server-negotiator.queue_time_us":27,"thread_start_us":82,"threads_started":1}
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:00 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:00.991371 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:00.987358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:48307)
0928 21:50:00.987406 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:00.987417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:00.987459 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:00.987563 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:00.987564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:00.987693 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:00.987724 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.987726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.988299 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.988300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.989091 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:00.989094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.989391 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.989392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:00.989444 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.989767 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:00.989771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:00.991303 (+ 1532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 21:50:00.991410 11987 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:00.987400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:48307 (local address 127.3.0.254:38607)
0928 21:50:00.987440 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:00.987442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:00.987451 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:50:00.987506 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:00.987507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:00.987515 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:00.987552 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:00.987780 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.988242 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.989159 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:00.989336 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:00.989381 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:00.991372 (+ 1991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:48307: BlockingRecv error: recv got EOF from 127.3.0.193:48307 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 21:50:00.991606 11984 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:50:01.055397 11988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:01.055598 11988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:01.055619 11988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:01.056721 11988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:01.056754 11988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:01.056779 11988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:50:01.058291 11988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.194.keytab
--principal=oryx/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:42395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=35291
--tserver_master_addrs=127.3.0.254:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:01.058605 11988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:01.058842 11988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:01.063500 11995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:01.063479 11994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:01.063515 11988 server_base.cc:1034] running on GCE node
W20260928 21:50:01.063486 11997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:01.063808 11988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:01.064033 11988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:01.065222 11988 hybrid_clock.cc:648] HybridClock initialized: now 1790632201065204 us; error 31 us; skew 500 ppm
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632201, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:01.067484 11988 init.cc:377] Logged in from keytab as oryx/127.3.0.194@KRBTEST.COM (short username oryx)
I20260928 21:50:01.067900 11988 webserver.cc:460] Webserver started at http://127.3.0.194:35291/ using document root <none> and password file <none>
I20260928 21:50:01.068068 11988 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:01.068114 11988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:01.069432 11988 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:50:01.069962 12004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:01.070115 11988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:01.070190 11988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "24f38b28f1374048a3be403b3ac69c49"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "4a776b327f54b059729385769b47e059"
server_key_iv: "cdf56e8240b83aa296f0fbaf4481ae60"
server_key_version: "encryptionkey@0"
I20260928 21:50:01.070504 11988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:01.084450 11988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:01.086125 11988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:01.086249 11988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:01.086447 11988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:01.086728 11988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:01.086766 11988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:01.086793 11988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:01.086813 11988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:01.092301 11988 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:42395
I20260928 21:50:01.092355 12117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:42395 every 8 connection(s)
I20260928 21:50:01.092681 11988 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:01.098050 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 11988
I20260928 21:50:01.098294 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:41107
--local_ip_for_outbound_sockets=127.3.0.195
--tserver_master_addrs=127.3.0.254:38607
--webserver_port=40801
--webserver_interface=127.3.0.195
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.0.195
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:50:01.099395 11987 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:01.093328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:50089 (local address 127.3.0.254:38607)
0928 21:50:01.093392 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:01.093394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:01.093593 (+ 199us) server_negotiation.cc:407] Connection header received
0928 21:50:01.094043 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:01.094044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:01.094057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:01.094105 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:01.094973 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.095450 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.096477 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.096674 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.096734 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.099340 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:50089: BlockingRecv error: recv got EOF from 127.3.0.194:50089 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:50:01.099433 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:01.093309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:50089)
0928 21:50:01.093481 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:01.093516 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:01.093970 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:01.094173 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:01.094177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:01.094717 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:01.094906 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.094911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.095543 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.095545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.096385 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.096388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.096766 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.096767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.096871 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.097288 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:01.097296 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:01.099242 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
W20260928 21:50:01.099730 12118 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:38607 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.0.254:38607: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:01.103924 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:01.099908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:45613)
0928 21:50:01.099966 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:01.099974 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:01.100009 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:01.100121 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:01.100122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:01.100234 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:01.100272 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.100274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.101077 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.101078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.101595 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.101598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.102017 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.102018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.102063 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.102460 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:01.102464 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:01.103881 (+ 1417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 21:50:01.104034 11987 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:01.099960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:45613 (local address 127.3.0.254:38607)
0928 21:50:01.100008 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:01.100009 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:01.100017 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:50:01.100059 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:01.100060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:01.100069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:01.100113 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:01.100334 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.101004 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.101677 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.101950 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.102016 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.103979 (+ 1963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:45613: BlockingRecv error: recv got EOF from 127.3.0.194:45613 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:01.108284 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:01.104137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:51711)
0928 21:50:01.104186 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:01.104198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:01.104248 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:01.104359 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:01.104360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:01.104515 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:01.104548 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.104549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.105109 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.105110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.105859 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.105862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.106162 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.106163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.106216 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.106545 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:01.106549 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:01.108240 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 21:50:01.108395 11987 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:01.104222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:51711 (local address 127.3.0.254:38607)
0928 21:50:01.104266 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:01.104268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:01.104278 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:50:01.104293 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:01.104294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:01.104301 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:01.104343 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:01.104611 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.105044 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.105934 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.106103 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.106141 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.108335 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:51711: BlockingRecv error: recv got EOF from 127.3.0.194:51711 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 21:50:01.108503 12118 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:50:01.177610 12121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:01.177821 12121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:01.177843 12121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:01.178936 12121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:01.178969 12121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:01.178994 12121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:50:01.180358 12121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/krb5kdc/oryx_127.3.0.195.keytab
--principal=oryx/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:41107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=40801
--tserver_master_addrs=127.3.0.254:38607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:01.180593 12121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:01.180792 12121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:01.185389 12126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:01.185410 12129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:01.185503 12121 server_base.cc:1034] running on GCE node
W20260928 21:50:01.185384 12127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:01.185827 12121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:01.186061 12121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:01.187202 12121 hybrid_clock.cc:648] HybridClock initialized: now 1790632201187188 us; error 29 us; skew 500 ppm
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632201, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:01.193939 12121 init.cc:377] Logged in from keytab as oryx/127.3.0.195@KRBTEST.COM (short username oryx)
I20260928 21:50:01.194396 12121 webserver.cc:460] Webserver started at http://127.3.0.195:40801/ using document root <none> and password file <none>
I20260928 21:50:01.194593 12121 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:01.194656 12121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:01.195904 12121 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:50:01.196434 12136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:01.196585 12121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:01.196674 12121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "77ec43f3eb5446cebb44f683ed9b71cd"
format_stamp: "Formatted at 2026-09-28 21:50:00 on dist-test-slave-jrck"
server_key: "d699e9e806c543f5a7caa54bf4a9ff3a"
server_key_iv: "4ca10a7452a8bfad14e14c264fe06857"
server_key_version: "encryptionkey@0"
I20260928 21:50:01.197011 12121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:01.208503 12121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:01.210107 12121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:01.210244 12121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:01.210451 12121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:01.210726 12121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:01.210759 12121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:01.210791 12121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:01.210817 12121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:01.216150 12121 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:41107
I20260928 21:50:01.216198 12249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:41107 every 8 connection(s)
I20260928 21:50:01.216553 12121 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:50:01.223153 11987 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:01.217180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:47663 (local address 127.3.0.254:38607)
0928 21:50:01.217227 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:01.217229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:01.217417 (+ 188us) server_negotiation.cc:407] Connection header received
0928 21:50:01.217909 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:01.217910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:01.217923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:01.217974 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:01.218874 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.219362 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.220321 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.220592 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.220649 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.223100 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:47663: BlockingRecv error: recv got EOF from 127.3.0.195:47663 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:50:01.223191 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 12121
I20260928 21:50:01.223199 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:01.217171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:47663)
0928 21:50:01.217301 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:01.217336 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:01.217809 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:01.218042 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:01.218046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:01.218587 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:01.218778 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.218782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.219443 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.219444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.220237 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.220240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.220664 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.220665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.220787 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.221182 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:01.221189 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:01.223009 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
W20260928 21:50:01.223431 12250 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:38607 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.0.254:38607: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:01.228080 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:01.223609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:38313)
0928 21:50:01.223651 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:01.223661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:01.223700 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:01.223905 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:01.223905 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:01.224051 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:01.224104 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.224106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.224846 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.224847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.225523 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.225525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.225833 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.225834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.225896 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.226260 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:01.226264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:01.228013 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:50:01.228168 11989 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:01.223710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:38313 (local address 127.3.0.254:38607)
0928 21:50:01.223775 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:01.223776 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:01.223783 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:50:01.223805 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:01.223806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:01.223817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:01.223883 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:01.224173 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.224785 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.225590 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.225777 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.225823 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.228106 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:38313: BlockingRecv error: recv got EOF from 127.3.0.195:38313 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:50:01.230561 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:01.223671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.3.0.254:38607)
0928 21:50:01.223710 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:01.223712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:01.223796 (+ 84us) server_negotiation.cc:407] Connection header received
0928 21:50:01.223830 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:01.223831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:01.223840 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:01.223887 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:01.224530 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.225050 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.225942 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.226099 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.226143 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.227358 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:01.227362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:01.227362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:50:01.227372 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:01.229318 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:01.229668 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:01.229671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:01.229671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:01.229772 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:01.229963 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:01.229963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:01.229964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:01.230163 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:01.230286 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:01.230373 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 21:50:01.230473 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":18}
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:01.232471 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:01.228292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:40537)
0928 21:50:01.228406 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:01.228414 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:01.228450 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:01.228598 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:01.228598 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:01.228704 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:01.228729 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.228730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.229536 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.229537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.230151 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.230153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.230594 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.230594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.230645 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.230986 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:01.230991 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:01.232406 (+ 1415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
W20260928 21:50:01.232564 11989 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:01.228347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:40537 (local address 127.3.0.254:38607)
0928 21:50:01.228390 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:01.228392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:01.228450 (+ 58us) server_negotiation.cc:407] Connection header received
0928 21:50:01.228522 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:01.228523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:01.228533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:01.228591 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:01.228787 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.229471 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.230220 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.230520 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.230583 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.232513 (+ 1930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:40537: BlockingRecv error: recv got EOF from 127.3.0.195:40537 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 21:50:01.232604 12250 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:01 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:01.996536 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:01.991972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:42925)
0928 21:50:01.992126 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:01.992138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:01.992210 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:01.992365 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:01.992366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:01.992559 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:01.992611 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.992613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.993221 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.993222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.993971 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:01.993975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.994300 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.994300 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:01.994337 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.994727 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:01.994731 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:01.996410 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
W20260928 21:50:01.996603 12256 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:01.992074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:42925 (local address 127.3.0.254:38607)
0928 21:50:01.992238 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:01.992239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:01.992249 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:50:01.992274 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:01.992274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:01.992285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:01.992354 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:01.992671 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.993155 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.994042 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:01.994245 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:01.994296 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:01.996539 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:42925: BlockingRecv error: recv got EOF from 127.3.0.193:42925 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
Sep 28 21:50:02 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:02 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:02.113276 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:02.108873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:41365)
0928 21:50:02.108993 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:02.109004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:02.109083 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:02.109206 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:02.109207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:02.109374 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:02.109420 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:02.109422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.110066 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.110067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:02.110631 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:02.110633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.110991 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.110992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:02.111045 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:02.111459 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:02.111465 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:02.113187 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
W20260928 21:50:02.113354 12256 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:02.108947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:41365 (local address 127.3.0.254:38607)
0928 21:50:02.109014 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:02.109016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:02.109068 (+ 52us) server_negotiation.cc:407] Connection header received
0928 21:50:02.109146 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:02.109147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:02.109158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:02.109202 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:02.109488 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.109988 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.110708 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.110934 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.110988 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:02.113300 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:41365: BlockingRecv error: recv got EOF from 127.3.0.194:41365 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 21:50:02 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:02 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:02.237377 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:02.232960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:39429)
0928 21:50:02.233126 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:02.233138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:02.233200 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:02.233327 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:02.233328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:02.233480 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:02.233528 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:02.233530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.234136 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.234137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:02.234688 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:02.234690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.235026 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.235027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:02.235076 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:02.235502 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:02.235506 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:02.237288 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":64,"threads_started":1}
W20260928 21:50:02.237440 12256 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:02.233067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:39429 (local address 127.3.0.254:38607)
0928 21:50:02.233113 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:02.233115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:02.233183 (+ 68us) server_negotiation.cc:407] Connection header received
0928 21:50:02.233254 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:02.233255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:02.233267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:02.233311 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:02.233576 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.234069 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.234757 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.234940 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.234993 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:02.237388 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:39429: BlockingRecv error: recv got EOF from 127.3.0.195:39429 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 21:50:02 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:03 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:03.001380 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:02.996999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:46033)
0928 21:50:02.997163 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:02.997176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:02.997239 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:02.997355 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:02.997355 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:02.997507 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:02.997560 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:02.997561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.998203 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.998204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:02.998764 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:02.998767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.999135 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.999135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:02.999178 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:02.999596 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:02.999601 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:03.001321 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
W20260928 21:50:03.001479 12260 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:02.997097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:46033 (local address 127.3.0.254:38607)
0928 21:50:02.997225 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:02.997226 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:02.997235 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:50:02.997287 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:02.997287 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:02.997297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:02.997351 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:02.997613 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.998129 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.998833 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:02.999072 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:02.999119 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:03.001411 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:46033: BlockingRecv error: recv got EOF from 127.3.0.193:46033 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
Sep 28 21:50:03 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:03 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:03.118059 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:03.113736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:56913)
0928 21:50:03.113882 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:03.113894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:03.113961 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:03.114084 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:03.114085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:03.114256 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:03.114302 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:03.114304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:03.114941 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:03.114942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:03.115474 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:03.115476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:03.115812 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:03.115812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:03.115864 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:03.116297 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:03.116301 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:03.117993 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
W20260928 21:50:03.118139 12260 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:03.113819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:56913 (local address 127.3.0.254:38607)
0928 21:50:03.113889 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:03.113891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:03.113951 (+ 60us) server_negotiation.cc:407] Connection header received
0928 21:50:03.114017 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:03.114018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:03.114030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:03.114080 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:03.114373 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:03.114870 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:03.115552 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:03.115753 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:03.115807 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:03.118081 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:56913: BlockingRecv error: recv got EOF from 127.3.0.194:56913 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 21:50:03 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:03 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:03.242651 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:03.237924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:41649)
0928 21:50:03.238088 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:03.238099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:03.238167 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:03.238383 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:03.238384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:03.238641 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:03.238690 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:03.238692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:03.239386 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:03.239387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:03.240020 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:03.240023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:03.240325 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:03.240325 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:03.240376 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:03.240851 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:03.240855 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:03.242574 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
W20260928 21:50:03.242731 12260 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:03.238074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:41649 (local address 127.3.0.254:38607)
0928 21:50:03.238133 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:03.238135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:03.238147 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:50:03.238249 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:03.238250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:03.238263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:03.238375 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:03.238797 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:03.239311 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:03.240083 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:03.240268 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:03.240320 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:03.242677 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:41649: BlockingRecv error: recv got EOF from 127.3.0.195:41649 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 21:50:04 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:04 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:04.005863 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:04.001806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:43641)
0928 21:50:04.001932 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:04.001942 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:04.001996 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:04.002123 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:04.002123 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:04.002275 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:04.002324 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:04.002326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.002929 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.002930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:04.003469 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:04.003471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.003767 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.003768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:04.003805 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:04.004184 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:04.004188 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:04.005793 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
W20260928 21:50:04.005944 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:04.001876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:43641 (local address 127.3.0.254:38607)
0928 21:50:04.001980 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:04.001982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:04.001991 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:50:04.002046 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:04.002046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:04.002057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:04.002114 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:04.002374 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.002874 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.003529 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.003714 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.003769 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:04.005880 (+ 2111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:43641: BlockingRecv error: recv got EOF from 127.3.0.193:43641 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
Sep 28 21:50:04 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:04 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:04.122522 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:04.118511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:34469)
0928 21:50:04.118627 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:04.118639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:04.118713 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:04.118853 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:04.118853 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:04.119007 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:04.119055 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:04.119056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.119641 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.119641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:04.120178 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:04.120179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.120482 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.120483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:04.120518 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:04.120919 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:04.120923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:04.122456 (+ 1533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
W20260928 21:50:04.122601 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:04.118590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:34469 (local address 127.3.0.254:38607)
0928 21:50:04.118661 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:04.118663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:04.118696 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:50:04.118780 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:04.118780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:04.118792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:04.118838 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:04.119101 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.119587 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.120235 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.120410 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.120479 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:04.122544 (+ 2065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:34469: BlockingRecv error: recv got EOF from 127.3.0.194:34469 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 21:50:04 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:04 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:04.247403 12266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:04.243151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:43191)
0928 21:50:04.243274 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:04.243285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:04.243347 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:04.243470 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:04.243471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:04.243638 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:04.243684 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:04.243687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.244319 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.244320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:04.244926 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:04.244929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.245257 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.245258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:04.245309 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:04.245685 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:04.245688 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:04.247307 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20260928 21:50:04.247453 12264 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:04.243212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:43191 (local address 127.3.0.254:38607)
0928 21:50:04.243269 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:04.243271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:04.243336 (+ 65us) server_negotiation.cc:407] Connection header received
0928 21:50:04.243405 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:04.243405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:04.243420 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:04.243467 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:04.243757 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.244247 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.245004 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:04.245198 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:04.245253 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:04.247399 (+ 2146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:43191: BlockingRecv error: recv got EOF from 127.3.0.195:43191 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 21:50:05 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:05 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:05.010902 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:05.006325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:41083)
0928 21:50:05.006447 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:05.006459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:05.006526 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:05.006721 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:05.006722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:05.006882 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:05.006930 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:05.006931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.007645 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.007646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:05.008195 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:05.008197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.008578 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.008579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:05.008637 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:05.009068 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:05.009072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:05.010825 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
W20260928 21:50:05.010979 12268 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:05.006400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:41083 (local address 127.3.0.254:38607)
0928 21:50:05.006545 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:05.006547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:05.006556 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:50:05.006580 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:05.006581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:05.006597 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:05.006669 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:05.007014 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.007549 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.008279 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.008501 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.008555 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:05.010915 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:41083: BlockingRecv error: recv got EOF from 127.3.0.193:41083 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
Sep 28 21:50:05 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:05 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:05.127267 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:05.122940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:52517)
0928 21:50:05.123107 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:05.123117 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:05.123171 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:05.123289 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:05.123289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:05.123440 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:05.123488 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:05.123489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.124099 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.124100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:05.124685 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:05.124688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.125034 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.125035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:05.125104 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:05.125545 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:05.125550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:05.127205 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
W20260928 21:50:05.127391 12268 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:05.123052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:52517 (local address 127.3.0.254:38607)
0928 21:50:05.123100 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:05.123102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:05.123160 (+ 58us) server_negotiation.cc:407] Connection header received
0928 21:50:05.123227 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:05.123227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:05.123238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:05.123275 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:05.123534 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.124027 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.124761 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.124945 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.124995 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:05.127320 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:52517: BlockingRecv error: recv got EOF from 127.3.0.194:52517 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 21:50:05 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:05 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:05.252540 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:05.247899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:46207)
0928 21:50:05.248116 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:05.248128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:05.248205 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:05.248346 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:05.248346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:05.248564 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:05.248619 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:05.248622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.249334 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.249334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:05.249902 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:05.249904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.250240 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.250241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:05.250291 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:05.250720 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:05.250725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:05.252432 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
W20260928 21:50:05.252612 12268 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:05.247986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:46207 (local address 127.3.0.254:38607)
0928 21:50:05.248088 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:05.248090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:05.248194 (+ 104us) server_negotiation.cc:407] Connection header received
0928 21:50:05.248270 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:05.248271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:05.248285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:05.248333 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:05.248693 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.249247 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.249986 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:05.250180 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:05.250236 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:05.252552 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:46207: BlockingRecv error: recv got EOF from 127.3.0.195:46207 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 21:50:06 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:06 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:06.020674 12271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:06.011363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:53655)
0928 21:50:06.011482 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:06.011494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:06.011558 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:06.011677 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:06.011678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:06.011856 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:06.011905 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:06.011907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.012530 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.012531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:06.013067 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:06.013069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.013447 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.013448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:06.013510 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:06.018806 (+ 5296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:06.018811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:06.020593 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
W20260928 21:50:06.020763 12272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:06.011418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:53655 (local address 127.3.0.254:38607)
0928 21:50:06.011553 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:06.011554 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:06.011564 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:50:06.011605 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:06.011606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:06.011616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:06.011663 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:06.011963 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.012480 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.013138 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.013380 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.013442 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:06.020690 (+ 7248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:53655: BlockingRecv error: recv got EOF from 127.3.0.193:53655 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
Sep 28 21:50:06 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:06 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:06.132017 12273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:06.127746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:34655)
0928 21:50:06.127881 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:06.127891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:06.127936 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:06.128055 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:06.128056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:06.128214 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:06.128263 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:06.128264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.128906 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.128906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:06.129433 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:06.129435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.129746 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.129748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:06.129801 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:06.130204 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:06.130208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:06.131959 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
W20260928 21:50:06.132091 12272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:06.127868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:34655 (local address 127.3.0.254:38607)
0928 21:50:06.127920 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:06.127921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:06.127930 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:50:06.127982 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:06.127983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:06.127994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:06.128041 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:06.128313 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.128843 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.129491 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.129664 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.129706 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:06.132033 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:34655: BlockingRecv error: recv got EOF from 127.3.0.194:34655 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 21:50:06 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:06 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:06.257270 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:06.253044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:59277)
0928 21:50:06.253178 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:06.253189 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:06.253244 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:06.253367 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:06.253368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:06.253522 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:06.253568 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:06.253569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.254243 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.254243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:06.254770 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:06.254772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.255056 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.255057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:06.255098 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:06.255484 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:06.255488 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:06.257187 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
W20260928 21:50:06.257357 12272 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:06.253137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:59277 (local address 127.3.0.254:38607)
0928 21:50:06.253224 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:06.253226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:06.253236 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:50:06.253294 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:06.253295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:06.253307 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:06.253354 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:06.253667 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.254178 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.254831 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:06.255006 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:06.255051 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:06.257291 (+ 2240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:59277: BlockingRecv error: recv got EOF from 127.3.0.195:59277 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 21:50:07 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:07 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:07.025528 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:07.021143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:55279)
0928 21:50:07.021275 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:07.021286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:07.021333 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:07.021456 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:07.021457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:07.021617 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:07.021665 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:07.021667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.022273 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.022274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:07.022799 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:07.022801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.023104 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.023105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:07.023141 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:07.023525 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:07.023530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:07.025447 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
W20260928 21:50:07.025614 12276 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:07.021218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:55279 (local address 127.3.0.254:38607)
0928 21:50:07.021325 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:07.021327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:07.021335 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:50:07.021388 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:07.021389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:07.021399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:07.021445 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:07.021720 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.022219 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.022864 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.023053 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.023100 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:07.025553 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:55279: BlockingRecv error: recv got EOF from 127.3.0.193:55279 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
Sep 28 21:50:07 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:07 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:07.136499 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:07.132476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:48635)
0928 21:50:07.132614 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:07.132623 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:07.132674 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:07.132790 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:07.132791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:07.132941 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:07.132989 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:07.132990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.133572 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.133572 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:07.134083 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:07.134085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.134364 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.134364 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:07.134401 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:07.134802 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:07.134806 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:07.136409 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
W20260928 21:50:07.136574 12276 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:07.132528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:48635 (local address 127.3.0.254:38607)
0928 21:50:07.132579 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:07.132580 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:07.132669 (+ 89us) server_negotiation.cc:407] Connection header received
0928 21:50:07.132716 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:07.132717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:07.132728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:07.132773 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:07.133036 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.133517 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.134140 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.134313 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.134355 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:07.136513 (+ 2158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:48635: BlockingRecv error: recv got EOF from 127.3.0.194:48635 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 21:50:07 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:07 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:07.262212 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:07.257689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:33209)
0928 21:50:07.257848 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:07.257859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:07.257936 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:07.258053 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:07.258054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:07.258228 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:07.258275 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:07.258277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.258970 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.258971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:07.259629 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:07.259632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.260052 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.260052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:07.260095 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:07.260539 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:07.260544 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:07.262133 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
W20260928 21:50:07.262298 12276 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:07.257784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:33209 (local address 127.3.0.254:38607)
0928 21:50:07.257849 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:07.257851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:07.257910 (+ 59us) server_negotiation.cc:407] Connection header received
0928 21:50:07.257988 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:07.257989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:07.258003 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:07.258049 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:07.258342 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.258890 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.259714 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:07.259999 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:07.260049 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:07.262235 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:33209: BlockingRecv error: recv got EOF from 127.3.0.195:33209 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 21:50:08 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:08 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:08.029994 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:08.025985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:44803)
0928 21:50:08.026106 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:08.026116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:08.026170 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:08.026296 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:08.026297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:08.026454 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:08.026502 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:08.026503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.027111 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.027112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:08.027643 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:08.027644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.027934 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.027934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:08.027970 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:08.028368 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:08.028372 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:08.029932 (+ 1560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
W20260928 21:50:08.030092 12280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:08.026077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:44803 (local address 127.3.0.254:38607)
0928 21:50:08.026181 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:08.026183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:08.026207 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:50:08.026227 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:08.026227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:08.026237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:08.026287 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:08.026554 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.027054 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.027700 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.027888 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.027930 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:08.030022 (+ 2092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:44803: BlockingRecv error: recv got EOF from 127.3.0.193:44803 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":45,"threads_started":1}
Sep 28 21:50:08 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:08 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:08.141383 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:08.136964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:41037)
0928 21:50:08.137090 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:08.137101 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:08.137159 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:08.137293 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:08.137294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:08.137447 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:08.137499 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:08.137501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.138231 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.138232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:08.138808 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:08.138810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.139145 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.139146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:08.139181 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:08.139612 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:08.139617 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:08.141307 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
W20260928 21:50:08.141456 12280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:08.137038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:41037 (local address 127.3.0.254:38607)
0928 21:50:08.137095 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:08.137097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:08.137155 (+ 58us) server_negotiation.cc:407] Connection header received
0928 21:50:08.137217 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:08.137218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:08.137228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:08.137266 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:08.137576 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.138171 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.138873 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.139083 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.139150 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:08.141404 (+ 2254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:41037: BlockingRecv error: recv got EOF from 127.3.0.194:41037 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 21:50:08 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:08 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:08.266837 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:08.262630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:43883)
0928 21:50:08.262797 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:08.262807 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:08.262861 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:08.262975 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:08.262976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:08.263127 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:08.263173 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:08.263175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.263782 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.263783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:08.264293 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:08.264294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.264596 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.264597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:08.264636 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:08.265010 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:08.265015 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:08.266756 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
W20260928 21:50:08.267037 12280 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:08.262697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:43883 (local address 127.3.0.254:38607)
0928 21:50:08.262757 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:08.262758 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:08.262856 (+ 98us) server_negotiation.cc:407] Connection header received
0928 21:50:08.262901 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:08.262902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:08.262915 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:08.262964 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:08.263227 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.263716 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.264355 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:08.264545 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:08.264592 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:08.266851 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:43883: BlockingRecv error: recv got EOF from 127.3.0.195:43883 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 21:50:09 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:09 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:09.034650 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:09.030443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:42373)
0928 21:50:09.030590 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:09.030603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:09.030667 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:09.030819 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:09.030820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:09.030982 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:09.031030 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:09.031032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.031637 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.031638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:09.032152 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:09.032153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.032478 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.032478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:09.032516 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:09.032928 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:09.032946 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:09.034588 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
W20260928 21:50:09.034754 12284 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:09.030575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:42373 (local address 127.3.0.254:38607)
0928 21:50:09.030689 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:09.030690 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:09.030697 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:50:09.030737 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:09.030738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:09.030748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:09.030808 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:09.031089 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.031583 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.032214 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.032403 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.032474 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:09.034676 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:42373: BlockingRecv error: recv got EOF from 127.3.0.193:42373 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
Sep 28 21:50:09 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:09 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:09.146339 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:09.141819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:56539)
0928 21:50:09.141935 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:09.141945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:09.142005 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:09.142123 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:09.142124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:09.142281 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:09.142325 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:09.142326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.143246 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.143247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:09.143771 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:09.143773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.144059 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.144060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:09.144095 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:09.144522 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:09.144526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:09.146250 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
W20260928 21:50:09.146396 12284 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:09.141858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:56539 (local address 127.3.0.254:38607)
0928 21:50:09.141931 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:09.141933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:09.141997 (+ 64us) server_negotiation.cc:407] Connection header received
0928 21:50:09.142056 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:09.142056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:09.142068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:09.142113 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:09.142386 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.143167 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.143834 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.144013 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.144054 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:09.146346 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:56539: BlockingRecv error: recv got EOF from 127.3.0.194:56539 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 21:50:09 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:09 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:09.271610 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:09.267262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:40833)
0928 21:50:09.267426 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:09.267436 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:09.267489 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:09.267624 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:09.267624 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:09.267771 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:09.267833 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:09.267834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.268526 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.268527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:09.269084 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:09.269087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.269409 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.269410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:09.269445 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:09.269859 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:09.269863 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:09.271552 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
W20260928 21:50:09.271711 12284 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:09.267340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:40833 (local address 127.3.0.254:38607)
0928 21:50:09.267386 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:09.267388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:09.267491 (+ 103us) server_negotiation.cc:407] Connection header received
0928 21:50:09.267552 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:09.267553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:09.267564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:09.267604 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:09.267921 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.268418 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.269153 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:09.269356 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:09.269406 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:09.271642 (+ 2236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:40833: BlockingRecv error: recv got EOF from 127.3.0.195:40833 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 21:50:10 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:10 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:10.039250 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:10.035080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:46521)
0928 21:50:10.035206 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:10.035217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:10.035300 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:10.035422 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:10.035422 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:10.035575 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:10.035624 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:10.035625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.036211 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.036211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:10.036798 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:10.036801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.037119 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.037120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:10.037157 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:10.037545 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:10.037550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:10.039192 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
W20260928 21:50:10.039383 12288 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:10.035140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:46521 (local address 127.3.0.254:38607)
0928 21:50:10.035253 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:10.035255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:10.035273 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:50:10.035348 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:10.035349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:10.035359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:10.035411 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:10.035667 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.036157 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.036871 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.037064 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.037116 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:10.039313 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:46521: BlockingRecv error: recv got EOF from 127.3.0.193:46521 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":49,"threads_started":1}
Sep 28 21:50:10 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:10 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:10.151208 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:10.146783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:50021)
0928 21:50:10.146935 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:10.146947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:10.147013 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:10.147142 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:10.147143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:10.147332 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:10.147377 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:10.147379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.148073 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.148074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:10.148631 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:10.148633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.148950 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.148951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:10.149001 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:10.149436 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:10.149440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:10.151131 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
W20260928 21:50:10.151280 12288 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:10.146865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:50021 (local address 127.3.0.254:38607)
0928 21:50:10.146922 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:10.146924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:10.147002 (+ 78us) server_negotiation.cc:407] Connection header received
0928 21:50:10.147072 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:10.147073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:10.147084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:10.147133 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:10.147451 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.148001 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.148708 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.148892 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.148947 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:10.151229 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:50021: BlockingRecv error: recv got EOF from 127.3.0.194:50021 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 21:50:10 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:10 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:10.276196 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:10.272019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:60583)
0928 21:50:10.272151 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:10.272162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:10.272204 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:10.272339 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:10.272339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:10.272510 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:10.272555 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:10.272557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.273188 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.273189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:10.273706 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:10.273708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.274036 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.274037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:10.274076 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:10.274467 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:10.274472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:10.276120 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
W20260928 21:50:10.276273 12288 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:10.272127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:60583 (local address 127.3.0.254:38607)
0928 21:50:10.272183 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:10.272185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:10.272195 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:50:10.272257 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:10.272258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:10.272268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:10.272327 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:10.272610 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.273116 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.273799 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:10.273982 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:10.274033 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:10.276210 (+ 2177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:60583: BlockingRecv error: recv got EOF from 127.3.0.195:60583 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 21:50:11 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:11 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:11.051085 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:11.039803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:57767)
0928 21:50:11.039931 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:11.039942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:11.039998 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:11.040133 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:11.040133 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:11.040328 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:11.040388 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:11.040391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.041209 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.041211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:11.041739 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:11.041740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.042075 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.042076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:11.042129 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:11.049282 (+ 7153us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:11.049288 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:11.051004 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
W20260928 21:50:11.051200 12292 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:11.039892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:57767 (local address 127.3.0.254:38607)
0928 21:50:11.040012 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:11.040014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:11.040023 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:50:11.040048 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:11.040048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:11.040058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:11.040122 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:11.040488 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.041139 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.041811 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.042017 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.042069 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:11.051125 (+ 9056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:57767: BlockingRecv error: recv got EOF from 127.3.0.193:57767 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
Sep 28 21:50:11 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:11 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:11.155720 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:11.151648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:46285)
0928 21:50:11.151783 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:11.151793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:11.151851 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:11.151962 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:11.151963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:11.152112 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:11.152159 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:11.152160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.152805 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.152806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:11.153318 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:11.153319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.153597 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.153598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:11.153633 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:11.154057 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:11.154061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:11.155660 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
W20260928 21:50:11.155812 12292 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:11.151742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:46285 (local address 127.3.0.254:38607)
0928 21:50:11.151786 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:11.151788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:11.151837 (+ 49us) server_negotiation.cc:407] Connection header received
0928 21:50:11.151905 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:11.151905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:11.151915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:11.151951 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:11.152209 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.152750 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.153377 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.153553 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.153595 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:11.155753 (+ 2158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:46285: BlockingRecv error: recv got EOF from 127.3.0.194:46285 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 21:50:11 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:11 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:11.280541 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:11.276629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:53439)
0928 21:50:11.276770 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:11.276781 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:11.276828 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:11.276939 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:11.276940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:11.277091 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:11.277134 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:11.277135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.277709 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.277709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:11.278238 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:11.278240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.278504 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.278504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:11.278540 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:11.278919 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:11.278924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:11.280478 (+ 1554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
W20260928 21:50:11.280632 12292 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:11.276714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:53439 (local address 127.3.0.254:38607)
0928 21:50:11.276804 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:11.276806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:11.276817 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:50:11.276874 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:11.276875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:11.276887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:11.276933 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:11.277177 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.277655 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.278293 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:11.278460 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:11.278500 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:11.280563 (+ 2063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:53439: BlockingRecv error: recv got EOF from 127.3.0.195:53439 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 21:50:12 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:12 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:12.056926 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:12.051556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:33631)
0928 21:50:12.051681 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:12.051694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:12.051763 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:12.051886 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:12.051887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:12.052043 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:12.052093 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:12.052095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.052905 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.052906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:12.053776 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:12.053780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.054221 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.054222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:12.054283 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:12.054760 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:12.054765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:12.056864 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
W20260928 21:50:12.057051 12296 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:12.051644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:33631 (local address 127.3.0.254:38607)
0928 21:50:12.051776 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:12.051777 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:12.051785 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:50:12.051809 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:12.051809 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:12.051820 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:12.051876 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:12.052145 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.052821 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.053855 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.054148 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.054207 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:12.056979 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:33631: BlockingRecv error: recv got EOF from 127.3.0.193:33631 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
Sep 28 21:50:12 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:12 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:12.160359 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:12.156131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:46271)
0928 21:50:12.156271 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:12.156282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:12.156351 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:12.156513 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:12.156514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:12.156685 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:12.156731 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:12.156732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.157368 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.157369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:12.157901 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:12.157902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.158260 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.158260 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:12.158312 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:12.158709 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:12.158713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:12.160284 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
W20260928 21:50:12.160480 12296 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:12.156206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:46271 (local address 127.3.0.254:38607)
0928 21:50:12.156277 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:12.156279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:12.156332 (+ 53us) server_negotiation.cc:407] Connection header received
0928 21:50:12.156415 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:12.156416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:12.156428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:12.156506 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:12.156796 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.157289 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.157970 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.158204 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.158261 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:12.160397 (+ 2136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:46271: BlockingRecv error: recv got EOF from 127.3.0.194:46271 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 21:50:12 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:12 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:12.285262 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:12.281006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:36999)
0928 21:50:12.281119 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:12.281131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:12.281203 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:12.281358 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:12.281359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:12.281529 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:12.281574 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:12.281575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.282205 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.282206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:12.282725 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:12.282727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.283044 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.283045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:12.283097 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:12.283515 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:12.283520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:12.285202 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
W20260928 21:50:12.285372 12296 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:12.281031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:36999 (local address 127.3.0.254:38607)
0928 21:50:12.281076 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:12.281079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:12.281183 (+ 104us) server_negotiation.cc:407] Connection header received
0928 21:50:12.281255 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:12.281256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:12.281272 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:12.281337 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:12.281644 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.282135 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.282800 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:12.282986 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:12.283039 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:12.285303 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:36999: BlockingRecv error: recv got EOF from 127.3.0.195:36999 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 21:50:13 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:13 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:13.061926 12299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:13.057544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:34263)
0928 21:50:13.057662 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:13.057672 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:13.057725 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:13.057844 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:13.057845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:13.058004 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:13.058056 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:13.058058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.058690 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.058691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:13.059284 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:13.059287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.059600 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.059600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:13.059638 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:13.060091 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:13.060096 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:13.061853 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260928 21:50:13.062005 12300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:13.057612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:34263 (local address 127.3.0.254:38607)
0928 21:50:13.057717 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:13.057719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:13.057727 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:50:13.057767 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:13.057767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:13.057777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:13.057834 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:13.058130 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.058628 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.059357 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.059551 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.059596 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:13.061935 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:34263: BlockingRecv error: recv got EOF from 127.3.0.193:34263 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
Sep 28 21:50:13 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:13 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:13.165077 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:13.160821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:48553)
0928 21:50:13.160936 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:13.160946 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:13.160999 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:13.161120 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:13.161120 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:13.161289 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:13.161336 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:13.161338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.161935 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.161936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:13.162461 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:13.162463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.162775 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.162776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:13.162828 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:13.163289 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:13.163294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:13.165002 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":51,"threads_started":1}
W20260928 21:50:13.165151 12300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:13.160890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:48553 (local address 127.3.0.254:38607)
0928 21:50:13.160942 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:13.160944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:13.160988 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:50:13.161058 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:13.161059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:13.161069 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:13.161106 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:13.161389 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.161882 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.162517 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.162692 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.162737 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:13.165095 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:48553: BlockingRecv error: recv got EOF from 127.3.0.194:48553 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 21:50:13 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:13 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:13.289848 12302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:13.285646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:55831)
0928 21:50:13.285757 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:13.285767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:13.285841 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:13.285943 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:13.285944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:13.286095 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:13.286142 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:13.286144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.286765 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.286767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:13.287314 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:13.287316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.287644 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.287645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:13.287689 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:13.288086 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:13.288091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:13.289769 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
W20260928 21:50:13.289959 12300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:13.285706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:55831 (local address 127.3.0.254:38607)
0928 21:50:13.285760 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:13.285762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:13.285826 (+ 64us) server_negotiation.cc:407] Connection header received
0928 21:50:13.285880 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:13.285880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:13.285891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:13.285932 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:13.286187 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.286678 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.287381 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:13.287562 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:13.287602 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:13.289876 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:55831: BlockingRecv error: recv got EOF from 127.3.0.195:55831 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 21:50:14 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:14 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:14.066505 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:14.062338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:44361)
0928 21:50:14.062465 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:14.062475 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:14.062537 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:14.062642 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:14.062643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:14.062789 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:14.062844 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:14.062846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.063434 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.063434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:14.063947 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:14.063948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.064237 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.064238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:14.064290 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:14.064708 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:14.064712 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:14.066396 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
W20260928 21:50:14.066557 12304 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:14.062395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:44361 (local address 127.3.0.254:38607)
0928 21:50:14.062491 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:14.062493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:14.062520 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:50:14.062576 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:14.062577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:14.062586 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:14.062639 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:14.062892 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.063382 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.064002 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.064190 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.064235 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:14.066494 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:44361: BlockingRecv error: recv got EOF from 127.3.0.193:44361 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":50,"threads_started":1}
Sep 28 21:50:14 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:14 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:14.169510 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:14.165510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:39389)
0928 21:50:14.165640 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:14.165650 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:14.165705 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:14.165844 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:14.165845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:14.165997 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:14.166046 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:14.166047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.166633 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.166633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:14.167158 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:14.167160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.167446 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.167447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:14.167481 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:14.167886 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:14.167890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:14.169440 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260928 21:50:14.169595 12304 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:14.165613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:39389 (local address 127.3.0.254:38607)
0928 21:50:14.165668 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:14.165670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:14.165690 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:50:14.165776 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:14.165776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:14.165788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:14.165833 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:14.166093 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.166579 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.167216 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.167397 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.167442 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:14.169537 (+ 2095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:39389: BlockingRecv error: recv got EOF from 127.3.0.194:39389 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 21:50:14 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:14 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:14.294623 12306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:14.290354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:43213)
0928 21:50:14.290489 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:14.290501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:14.290563 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:14.290677 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:14.290678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:14.290845 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:14.290889 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:14.290890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.291511 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.291512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:14.292122 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:14.292126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.292444 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.292445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:14.292517 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:14.292907 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:14.292911 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:14.294553 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
W20260928 21:50:14.294760 12304 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:14.290435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:43213 (local address 127.3.0.254:38607)
0928 21:50:14.290493 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:14.290495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:14.290544 (+ 49us) server_negotiation.cc:407] Connection header received
0928 21:50:14.290609 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:14.290610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:14.290625 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:14.290668 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:14.290951 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.291439 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.292199 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:14.292386 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:14.292439 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:14.294692 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:43213: BlockingRecv error: recv got EOF from 127.3.0.195:43213 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 28 21:50:15 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:15 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:15.071313 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:15.066937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:37861)
0928 21:50:15.067091 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:15.067101 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:15.067158 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:15.067298 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:15.067298 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:15.067454 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:15.067502 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:15.067504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.068314 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.068314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:15.068898 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:15.068901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.069190 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.069191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:15.069227 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:15.069662 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:15.069667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:15.071245 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
W20260928 21:50:15.071408 12308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:15.067001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:37861 (local address 127.3.0.254:38607)
0928 21:50:15.067110 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:15.067112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:15.067143 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:50:15.067215 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:15.067216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:15.067227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:15.067275 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:15.067553 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.068249 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.068958 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.069144 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.069187 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:15.071335 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:37861: BlockingRecv error: recv got EOF from 127.3.0.193:37861 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
Sep 28 21:50:15 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:15 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:15.174291 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:15.169966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:56255)
0928 21:50:15.170105 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:15.170116 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:15.170163 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:15.170287 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:15.170287 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:15.170450 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:15.170497 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:15.170498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.171142 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.171142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:15.171667 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:15.171669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.171975 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.171976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:15.172012 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:15.172400 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:15.172404 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:15.174205 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
W20260928 21:50:15.174374 12308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:15.170068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:56255 (local address 127.3.0.254:38607)
0928 21:50:15.170141 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:15.170143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:15.170153 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:50:15.170212 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:15.170212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:15.170223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:15.170275 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:15.170562 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.171080 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.171733 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.171921 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.171972 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:15.174308 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:56255: BlockingRecv error: recv got EOF from 127.3.0.194:56255 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 21:50:15 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:15 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.195@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:15.299177 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:15.295112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.195:46171)
0928 21:50:15.295220 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:15.295231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:15.295290 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:15.295417 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:15.295417 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:15.295570 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:15.295615 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:15.295617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.296215 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.296215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:15.296788 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:15.296791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.297097 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.297097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:15.297133 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:15.297514 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:15.297518 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:15.299111 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
W20260928 21:50:15.299248 12308 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:15.295170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:46171 (local address 127.3.0.254:38607)
0928 21:50:15.295247 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:15.295249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:15.295278 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:50:15.295346 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:15.295347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:15.295358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:15.295404 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:15.295673 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.296161 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.296864 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:15.297042 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:15.297094 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:15.299187 (+ 2093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.195:46171: BlockingRecv error: recv got EOF from 127.3.0.195:46171 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 21:50:16 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:16 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.193@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:16.079573 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:16.071807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.193:48603)
0928 21:50:16.071930 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:16.071939 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:16.071979 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:16.072120 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:16.072121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:16.072287 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:16.072334 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:16.072336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.073023 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.073024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:16.073620 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:16.073623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.073940 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.073941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:16.073984 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.077879 (+ 3895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:16.077884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:16.079501 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":54,"threads_started":1}
W20260928 21:50:16.079676 12312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:16.071885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:48603 (local address 127.3.0.254:38607)
0928 21:50:16.072007 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:16.072009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:16.072015 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:50:16.072037 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:16.072038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:16.072048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:16.072107 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:16.072386 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.072961 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.073682 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.073884 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.073929 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.079599 (+ 5670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.193:48603: BlockingRecv error: recv got EOF from 127.3.0.193:48603 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
Sep 28 21:50:16 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:50:16 dist-test-slave-jrck krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.0.194@KRBTEST.COM for oryx/127.3.0.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:50:16.179056 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:16.174822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38607 (local address 127.3.0.194:47719)
0928 21:50:16.174966 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:16.174976 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:16.175034 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:16.175159 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:16.175159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:16.175319 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:16.175367 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:16.175368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.175968 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.175969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:16.176543 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:16.176544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.176829 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.176829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:16.176865 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.177350 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:16.177353 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:16.178968 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.0.254:38607: Server oryx/127.3.0.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
W20260928 21:50:16.179113 12312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:16.174912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:47719 (local address 127.3.0.254:38607)
0928 21:50:16.174968 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:16.174970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:16.175026 (+ 56us) server_negotiation.cc:407] Connection header received
0928 21:50:16.175077 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:16.175077 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:16.175087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:16.175134 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:16.175419 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.175913 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.176598 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.176780 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.176840 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.179061 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.0.194:47719: BlockingRecv error: recv got EOF from 127.3.0.194:47719 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:50:16.224411 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 11836
I20260928 21:50:16.228291 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 11988
I20260928 21:50:16.231913 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 12121
I20260928 21:50:16.235357 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 11766
2026-09-28T21:50:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17288 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T21:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:50:16Z Disabled control of system clock
I20260928 21:50:16.259895 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:41801
--webserver_interface=127.3.0.254
--webserver_port=0
--builtin_ntp_servers=127.3.0.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:41801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:50:16.337661 12317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:16.337863 12317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:16.337884 12317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:16.339054 12317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:50:16.339092 12317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:16.339103 12317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:16.339116 12317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:50:16.339140 12317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:50:16.340435 12317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:41801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:16.340705 12317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:16.340994 12317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:16.345666 12322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:16.345674 12317 server_base.cc:1034] running on GCE node
W20260928 21:50:16.345623 12325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:16.345628 12323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:16.346139 12317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:16.346415 12317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:16.347550 12317 hybrid_clock.cc:648] HybridClock initialized: now 1790632216347533 us; error 30 us; skew 500 ppm
I20260928 21:50:16.348690 12317 webserver.cc:460] Webserver started at http://127.3.0.254:43817/ using document root <none> and password file <none>
I20260928 21:50:16.348915 12317 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:16.348964 12317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:16.349086 12317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:16.350096 12317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "f1a025149d5f42bd8937e27ae3908c76"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "713c35a51cd7d2f1ff26c975b7a69d05"
server_key_iv: "da1f9f999373888d4c70aa49fd3d9ecb"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.350414 12317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "f1a025149d5f42bd8937e27ae3908c76"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "713c35a51cd7d2f1ff26c975b7a69d05"
server_key_iv: "da1f9f999373888d4c70aa49fd3d9ecb"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.351532 12317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:50:16.352137 12331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.352303 12317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:16.352383 12317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "f1a025149d5f42bd8937e27ae3908c76"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "713c35a51cd7d2f1ff26c975b7a69d05"
server_key_iv: "da1f9f999373888d4c70aa49fd3d9ecb"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.352442 12317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:16.367724 12317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:16.368003 12317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:16.368142 12317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:16.371842 12317 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:41801
I20260928 21:50:16.371871 12383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:41801 every 8 connection(s)
I20260928 21:50:16.372193 12317 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:50:16.372862 12384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:50:16.373649 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 12317
I20260928 21:50:16.373729 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:50:16.373903 3075 external_mini_cluster.cc:1428] Setting key 5b161f8f36fdf8dbd50ce35f9d8cb72f
I20260928 21:50:16.375698 12384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76: Bootstrap starting.
I20260928 21:50:16.376274 12384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76: Neither blocks nor log segments found. Creating new log.
I20260928 21:50:16.376555 12384 log.cc:824] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76: Log is configured to *not* fsync() on all Append() calls
I20260928 21:50:16.377164 12384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76: No bootstrap required, opened a new log
I20260928 21:50:16.379015 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:16.374413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.3.0.254:41801)
0928 21:50:16.374612 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:16.374620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:16.374644 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:50:16.375060 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:16.375067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:16.375127 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:16.375253 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:16.375504 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.376503 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.377271 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.377686 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.377793 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.378445 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:16.378449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:16.378449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:50:16.378607 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:16.378638 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:50:16.378649 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:16.378708 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:16.378773 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:50:16.378856 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260928 21:50:16.379002 12384 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a025149d5f42bd8937e27ae3908c76" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 41801 } }
I20260928 21:50:16.379175 12384 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:50:16.379217 12384 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a025149d5f42bd8937e27ae3908c76, State: Initialized, Role: FOLLOWER
I20260928 21:50:16.379338 12384 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a025149d5f42bd8937e27ae3908c76" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 41801 } }
I20260928 21:50:16.379405 12384 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:50:16.379433 12384 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:50:16.379479 12384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:50:16.380014 12384 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a025149d5f42bd8937e27ae3908c76" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 41801 } }
I20260928 21:50:16.380112 12384 leader_election.cc:302] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1a025149d5f42bd8937e27ae3908c76; no voters:
I20260928 21:50:16.380263 12384 leader_election.cc:288] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:50:16.380332 12389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:50:16.380497 12389 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [term 1 LEADER]: Becoming Leader. State: Replica: f1a025149d5f42bd8937e27ae3908c76, State: Running, Role: LEADER
I20260928 21:50:16.380530 12384 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:50:16.380585 12389 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a025149d5f42bd8937e27ae3908c76" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 41801 } }
I20260928 21:50:16.380964 12390 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1a025149d5f42bd8937e27ae3908c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a025149d5f42bd8937e27ae3908c76" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 41801 } } }
I20260928 21:50:16.380960 12391 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a025149d5f42bd8937e27ae3908c76. Latest consensus state: current_term: 1 leader_uuid: "f1a025149d5f42bd8937e27ae3908c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a025149d5f42bd8937e27ae3908c76" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 41801 } } }
I20260928 21:50:16.381057 12390 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [sys.catalog]: This master's current role is: LEADER
I20260928 21:50:16.381062 12391 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76 [sys.catalog]: This master's current role is: LEADER
I20260928 21:50:16.381309 12398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:50:16.381781 12398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:50:16.383249 12398 catalog_manager.cc:1347] Generated new cluster ID: 6ff4e0030a3f4ddeb4f4036390d93ebf
I20260928 21:50:16.383294 12398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:50:16.403929 12398 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:50:16.404500 12398 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:50:16.406711 12398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1a025149d5f42bd8937e27ae3908c76: Generated new TSK 0
I20260928 21:50:16.406903 12398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:50:16.409091 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:41801
--builtin_ntp_servers=127.3.0.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 21:50:16.499778 12408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:16.499979 12408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:16.499998 12408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:16.501207 12408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:16.501255 12408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:16.501286 12408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:50:16.502612 12408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:16.502842 12408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:16.503130 12408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:16.507831 12414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:16.507887 12416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:16.507893 12413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:16.508374 12408 server_base.cc:1034] running on GCE node
I20260928 21:50:16.508582 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:16.508803 12408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:16.509948 12408 hybrid_clock.cc:648] HybridClock initialized: now 1790632216509934 us; error 32 us; skew 500 ppm
I20260928 21:50:16.511116 12408 webserver.cc:460] Webserver started at http://127.3.0.193:37427/ using document root <none> and password file <none>
I20260928 21:50:16.511358 12408 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:16.511401 12408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:16.511525 12408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:16.512612 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "0e8e4337fed249cdadd5e28f720f1f02"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "aa60d29d7873972268dd0ad6298b7179"
server_key_iv: "1223d4d6a29619b4d78f0164421da2d3"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.512925 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "0e8e4337fed249cdadd5e28f720f1f02"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "aa60d29d7873972268dd0ad6298b7179"
server_key_iv: "1223d4d6a29619b4d78f0164421da2d3"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.514112 12408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:50:16.514796 12422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.514945 12408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:50:16.515010 12408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "0e8e4337fed249cdadd5e28f720f1f02"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "aa60d29d7873972268dd0ad6298b7179"
server_key_iv: "1223d4d6a29619b4d78f0164421da2d3"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.515079 12408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:16.530506 12408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:16.530807 12408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:16.530936 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:16.531140 12408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:16.531459 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:16.531494 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.531522 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:16.531545 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.537086 12408 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:39227
I20260928 21:50:16.537127 12535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:39227 every 8 connection(s)
I20260928 21:50:16.537441 12408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:50:16.542434 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:16.538104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:57703 (local address 127.3.0.254:41801)
0928 21:50:16.538163 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:16.538164 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:16.538390 (+ 226us) server_negotiation.cc:407] Connection header received
0928 21:50:16.538918 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:16.538920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:16.538932 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:16.538985 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:16.539627 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.540124 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.541155 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.541354 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.541411 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.542133 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:16.542134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:16.542134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:50:16.542139 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:16.542155 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:50:16.542159 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:16.542202 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:16.542279 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 21:50:16.542355 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:50:16.542572 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:16.538081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:41801 (local address 127.3.0.193:57703)
0928 21:50:16.538271 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:16.538308 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:16.538820 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:16.539056 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:16.539060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:16.539070 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:16.539538 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:16.539543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.540209 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.540210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:16.541073 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:16.541075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.541424 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.541425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:16.541552 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.542008 (+ 456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:50:16.542008 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:16.542037 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:50:16.542058 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:16.542207 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:16.542213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:16.542216 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:50:16.542274 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:50:16.542375 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260928 21:50:16.543008 12536 heartbeater.cc:347] Connected to a master server at 127.3.0.254:41801
I20260928 21:50:16.543097 12536 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:16.543270 12536 heartbeater.cc:511] Master 127.3.0.254:41801 requested a full tablet report, sending...
I20260928 21:50:16.543732 12348 ts_manager.cc:194] Registered new tserver with Master: 0e8e4337fed249cdadd5e28f720f1f02 (127.3.0.193:39227)
I20260928 21:50:16.544317 12348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.193:57703
I20260928 21:50:16.544397 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 12408
I20260928 21:50:16.544499 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:50:16.544695 3075 external_mini_cluster.cc:1428] Setting key 804af8b75259bd0842f720fc03a15b53
I20260928 21:50:16.545493 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:41801
--builtin_ntp_servers=127.3.0.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 21:50:16.623551 12539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:16.623746 12539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:16.623766 12539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:16.624938 12539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:16.624976 12539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:16.625001 12539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:50:16.626304 12539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:16.626525 12539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:16.626811 12539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:16.631445 12545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:16.631474 12544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:16.631488 12547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:16.631518 12539 server_base.cc:1034] running on GCE node
I20260928 21:50:16.631830 12539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:16.632066 12539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:16.633203 12539 hybrid_clock.cc:648] HybridClock initialized: now 1790632216633187 us; error 29 us; skew 500 ppm
I20260928 21:50:16.634397 12539 webserver.cc:460] Webserver started at http://127.3.0.194:42475/ using document root <none> and password file <none>
I20260928 21:50:16.634617 12539 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:16.634665 12539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:16.634809 12539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:16.635831 12539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "db17bad934d448e7b6431bb0bf0651ad"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "3e8a7474e984ec8d4f13a59548e845fa"
server_key_iv: "f1a36c4cb981ba81369cbcc4fcf8b00e"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.636148 12539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "db17bad934d448e7b6431bb0bf0651ad"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "3e8a7474e984ec8d4f13a59548e845fa"
server_key_iv: "f1a36c4cb981ba81369cbcc4fcf8b00e"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.637352 12539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:50:16.638031 12553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.638173 12539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:16.638252 12539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "db17bad934d448e7b6431bb0bf0651ad"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "3e8a7474e984ec8d4f13a59548e845fa"
server_key_iv: "f1a36c4cb981ba81369cbcc4fcf8b00e"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.638311 12539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:16.664814 12539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:16.665128 12539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:16.665261 12539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:16.665484 12539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:16.665802 12539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:16.665835 12539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.665863 12539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:16.665881 12539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.671286 12539 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:45487
I20260928 21:50:16.671342 12666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:45487 every 8 connection(s)
I20260928 21:50:16.671651 12539 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:50:16.676330 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:16.672345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:56747 (local address 127.3.0.254:41801)
0928 21:50:16.672526 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:16.672528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:16.672646 (+ 118us) server_negotiation.cc:407] Connection header received
0928 21:50:16.673124 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:16.673125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:16.673135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:16.673197 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:16.673797 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.674274 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.675220 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.675399 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.675444 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.676118 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:16.676119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:16.676120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:50:16.676123 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:16.676137 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:50:16.676140 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:16.676179 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:16.676241 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:50:16.676265 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260928 21:50:16.676528 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:16.672292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:41801 (local address 127.3.0.194:56747)
0928 21:50:16.672507 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:16.672557 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:16.673048 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:16.673260 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:16.673264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:16.673272 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:16.673735 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:16.673739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.674342 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.674343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:16.675156 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:16.675158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.675450 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.675451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:16.675553 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.676000 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:50:16.676000 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:16.676028 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:50:16.676050 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:16.676184 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:16.676187 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:16.676189 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:50:16.676235 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:50:16.676311 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260928 21:50:16.676991 12667 heartbeater.cc:347] Connected to a master server at 127.3.0.254:41801
I20260928 21:50:16.677080 12667 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:16.677246 12667 heartbeater.cc:511] Master 127.3.0.254:41801 requested a full tablet report, sending...
I20260928 21:50:16.677582 12348 ts_manager.cc:194] Registered new tserver with Master: db17bad934d448e7b6431bb0bf0651ad (127.3.0.194:45487)
I20260928 21:50:16.677930 12348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.194:56747
I20260928 21:50:16.679883 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 12539
I20260928 21:50:16.679950 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:50:16.680094 3075 external_mini_cluster.cc:1428] Setting key 14a05e5ec3aec6a765398fbf62c26fd0
I20260928 21:50:16.680959 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:41801
--builtin_ntp_servers=127.3.0.212:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 21:50:16.759920 12670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:16.760119 12670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:16.760138 12670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:16.761277 12670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:16.761312 12670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:16.761333 12670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:50:16.762606 12670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:41801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:16.762833 12670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:16.763087 12670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:16.767745 12675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:16.767786 12678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:16.767781 12670 server_base.cc:1034] running on GCE node
W20260928 21:50:16.767769 12676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:16.768158 12670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:16.768369 12670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:16.769515 12670 hybrid_clock.cc:648] HybridClock initialized: now 1790632216769498 us; error 33 us; skew 500 ppm
I20260928 21:50:16.770680 12670 webserver.cc:460] Webserver started at http://127.3.0.195:34447/ using document root <none> and password file <none>
I20260928 21:50:16.770905 12670 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:16.770951 12670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:16.771078 12670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:16.772107 12670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "232b46431b8f4d73a16032f9aa5c5acd"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "a92c03040f189f75ca73748abf80629d"
server_key_iv: "e0ed1c1419cd0521bbe89bab5db89729"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.772428 12670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "232b46431b8f4d73a16032f9aa5c5acd"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "a92c03040f189f75ca73748abf80629d"
server_key_iv: "e0ed1c1419cd0521bbe89bab5db89729"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.773653 12670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:50:16.774318 12684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.774457 12670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:16.774531 12670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "232b46431b8f4d73a16032f9aa5c5acd"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "a92c03040f189f75ca73748abf80629d"
server_key_iv: "e0ed1c1419cd0521bbe89bab5db89729"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.774595 12670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:16.799177 12670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:16.799481 12670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:16.799608 12670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:16.799839 12670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:16.800164 12670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:16.800196 12670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.800217 12670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:16.800228 12670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.805584 12670 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:35043
I20260928 21:50:16.805639 12797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:35043 every 8 connection(s)
I20260928 21:50:16.805950 12670 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:50:16.810774 12387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:16.806620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:51207 (local address 127.3.0.254:41801)
0928 21:50:16.806702 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:16.806705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:16.806976 (+ 271us) server_negotiation.cc:407] Connection header received
0928 21:50:16.807497 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:16.807499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:16.807509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:16.807561 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:16.808153 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.808698 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.809612 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.809801 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.809848 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.810552 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:16.810554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:16.810554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:50:16.810558 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:16.810574 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:50:16.810579 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:16.810628 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:16.810691 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 21:50:16.810712 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:50:16.810935 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:16.806607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:41801 (local address 127.3.0.195:51207)
0928 21:50:16.806853 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:16.806889 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:16.807407 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:16.807616 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:16.807620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:16.807628 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:16.808085 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:16.808089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.808764 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.808765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:16.809544 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:16.809547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:16.809860 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:16.809860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:16.809961 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:16.810406 (+ 445us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:50:16.810407 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:16.810433 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:50:16.810455 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:16.810632 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:16.810635 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:16.810637 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:50:16.810683 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:50:16.810757 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":144,"threads_started":1}
I20260928 21:50:16.811367 12798 heartbeater.cc:347] Connected to a master server at 127.3.0.254:41801
I20260928 21:50:16.811455 12798 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:16.811625 12798 heartbeater.cc:511] Master 127.3.0.254:41801 requested a full tablet report, sending...
I20260928 21:50:16.811956 12348 ts_manager.cc:194] Registered new tserver with Master: 232b46431b8f4d73a16032f9aa5c5acd (127.3.0.195:35043)
I20260928 21:50:16.812306 12348 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.195:51207
I20260928 21:50:16.815492 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 12670
I20260928 21:50:16.815562 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:50:16.815713 3075 external_mini_cluster.cc:1428] Setting key 8306292e2532b55fe0595ea095aa48b7
I20260928 21:50:16.816437 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:50:16.818064 12387 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:16.817620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.3.0.254:41801)
0928 21:50:16.817685 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:16.817687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:16.817781 (+ 94us) server_negotiation.cc:407] Connection header received
0928 21:50:16.817812 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:16.817813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:16.817821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:16.817861 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:16.818005 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44636: BlockingRecv error: recv got EOF from 127.0.0.1:44636 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:50:16.818120 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 12408
I20260928 21:50:16.822741 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 12539
I20260928 21:50:16.826224 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 12670
I20260928 21:50:16.829641 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 12317
2026-09-28T21:50:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (593 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T21:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:50:16Z Disabled control of system clock
I20260928 21:50:16.854153 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:39385
--webserver_interface=127.3.0.254
--webserver_port=0
--builtin_ntp_servers=127.3.0.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:39385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:50:16.931603 12809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:16.931806 12809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:16.931833 12809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:16.932992 12809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:50:16.933032 12809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:16.933043 12809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:16.933054 12809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:50:16.933065 12809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:50:16.934374 12809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:39385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:39385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:16.934600 12809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:16.934870 12809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:16.939154 12817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:16.939250 12815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:16.939188 12814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:16.939469 12809 server_base.cc:1034] running on GCE node
I20260928 21:50:16.939702 12809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:16.939960 12809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:16.941090 12809 hybrid_clock.cc:648] HybridClock initialized: now 1790632216941081 us; error 26 us; skew 500 ppm
I20260928 21:50:16.942176 12809 webserver.cc:460] Webserver started at http://127.3.0.254:46039/ using document root <none> and password file <none>
I20260928 21:50:16.942379 12809 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:16.942427 12809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:16.942545 12809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:16.943537 12809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "4bf042f99fec444abe00b456a4244558"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "cfde3554d71e06a2bd927492ae21c496"
server_key_iv: "cce1acfdfb2810b37e0220caf8032403"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.943876 12809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "4bf042f99fec444abe00b456a4244558"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "cfde3554d71e06a2bd927492ae21c496"
server_key_iv: "cce1acfdfb2810b37e0220caf8032403"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.945082 12809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:50:16.945683 12823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:16.945843 12809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:16.945921 12809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "4bf042f99fec444abe00b456a4244558"
format_stamp: "Formatted at 2026-09-28 21:50:16 on dist-test-slave-jrck"
server_key: "cfde3554d71e06a2bd927492ae21c496"
server_key_iv: "cce1acfdfb2810b37e0220caf8032403"
server_key_version: "encryptionkey@0"
I20260928 21:50:16.945978 12809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:16.949160 12809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:16.949409 12809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:16.949524 12809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:16.953387 12809 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:39385
I20260928 21:50:16.953423 12875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:39385 every 8 connection(s)
I20260928 21:50:16.953711 12809 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:50:16.954310 12876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:50:16.956910 12876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558: Bootstrap starting.
I20260928 21:50:16.957403 12876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558: Neither blocks nor log segments found. Creating new log.
I20260928 21:50:16.957644 12876 log.cc:824] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558: Log is configured to *not* fsync() on all Append() calls
I20260928 21:50:16.957680 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 12809
I20260928 21:50:16.957759 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:50:16.957943 3075 external_mini_cluster.cc:1428] Setting key e5f41f7efd342c8897b85eb8840beebc
I20260928 21:50:16.958230 12876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558: No bootstrap required, opened a new log
I20260928 21:50:16.960112 12876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf042f99fec444abe00b456a4244558" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39385 } }
I20260928 21:50:16.960243 12876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:50:16.960269 12876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf042f99fec444abe00b456a4244558, State: Initialized, Role: FOLLOWER
I20260928 21:50:16.960408 12876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf042f99fec444abe00b456a4244558" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39385 } }
I20260928 21:50:16.960523 12876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:50:16.960588 12876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:50:16.960650 12876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:50:16.960688 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:16.958383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.3.0.254:39385)
0928 21:50:16.958573 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:16.958581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:16.958604 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:50:16.959352 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:16.959359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:16.959397 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:16.959519 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:16.960171 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:16.960174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:16.960174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:50:16.960178 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:16.960214 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:50:16.960223 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:16.960282 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:16.960392 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 21:50:16.960449 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260928 21:50:16.961539 12876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf042f99fec444abe00b456a4244558" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39385 } }
I20260928 21:50:16.961632 12876 leader_election.cc:302] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bf042f99fec444abe00b456a4244558; no voters:
I20260928 21:50:16.961777 12876 leader_election.cc:288] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:50:16.961814 12881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:50:16.961946 12881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [term 1 LEADER]: Becoming Leader. State: Replica: 4bf042f99fec444abe00b456a4244558, State: Running, Role: LEADER
I20260928 21:50:16.961998 12876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:50:16.962031 12881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf042f99fec444abe00b456a4244558" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39385 } }
I20260928 21:50:16.962370 12881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bf042f99fec444abe00b456a4244558. Latest consensus state: current_term: 1 leader_uuid: "4bf042f99fec444abe00b456a4244558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf042f99fec444abe00b456a4244558" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39385 } } }
I20260928 21:50:16.962515 12881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [sys.catalog]: This master's current role is: LEADER
I20260928 21:50:16.962401 12883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bf042f99fec444abe00b456a4244558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf042f99fec444abe00b456a4244558" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39385 } } }
I20260928 21:50:16.962756 12883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558 [sys.catalog]: This master's current role is: LEADER
I20260928 21:50:16.962756 12890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:50:16.963263 12890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:50:16.964699 12890 catalog_manager.cc:1347] Generated new cluster ID: cbdb8e71ce9f4608800b903ff2d34324
I20260928 21:50:16.964745 12890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:50:16.987095 12890 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:50:16.987262 12890 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:50:16.992844 12890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bf042f99fec444abe00b456a4244558: Generated new TSK 0
I20260928 21:50:16.993024 12890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:50:16.995958 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39385
--builtin_ntp_servers=127.3.0.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:50:17.109483 12900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:17.109781 12900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:17.109817 12900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:17.111479 12900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:17.111534 12900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:17.111572 12900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:50:17.113576 12900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:17.113838 12900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:17.114179 12900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:17.119163 12905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:17.119163 12908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:17.119166 12906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:17.119537 12900 server_base.cc:1034] running on GCE node
I20260928 21:50:17.119721 12900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:17.119953 12900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:17.121100 12900 hybrid_clock.cc:648] HybridClock initialized: now 1790632217121088 us; error 27 us; skew 500 ppm
I20260928 21:50:17.122409 12900 webserver.cc:460] Webserver started at http://127.3.0.193:44547/ using document root <none> and password file <none>
I20260928 21:50:17.122653 12900 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:17.122707 12900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:17.122829 12900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:17.124140 12900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "e659ea6abf1641fe8c54fc70d3f68f45"
format_stamp: "Formatted at 2026-09-28 21:50:17 on dist-test-slave-jrck"
server_key: "ceb0e95a0deef5c4edd12d48648d1cf1"
server_key_iv: "fdea988cdcc64d40374275d8e2d17c68"
server_key_version: "encryptionkey@0"
I20260928 21:50:17.124555 12900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "e659ea6abf1641fe8c54fc70d3f68f45"
format_stamp: "Formatted at 2026-09-28 21:50:17 on dist-test-slave-jrck"
server_key: "ceb0e95a0deef5c4edd12d48648d1cf1"
server_key_iv: "fdea988cdcc64d40374275d8e2d17c68"
server_key_version: "encryptionkey@0"
I20260928 21:50:17.126062 12900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:50:17.126899 12914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:17.127045 12900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:17.127128 12900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "e659ea6abf1641fe8c54fc70d3f68f45"
format_stamp: "Formatted at 2026-09-28 21:50:17 on dist-test-slave-jrck"
server_key: "ceb0e95a0deef5c4edd12d48648d1cf1"
server_key_iv: "fdea988cdcc64d40374275d8e2d17c68"
server_key_version: "encryptionkey@0"
I20260928 21:50:17.127197 12900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:17.130393 12900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:17.130622 12900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:17.130739 12900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:17.130927 12900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:17.131220 12900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:17.131253 12900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:17.131279 12900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:17.131304 12900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:17.136477 12900 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:45399
I20260928 21:50:17.136525 13027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:45399 every 8 connection(s)
I20260928 21:50:17.136821 12900 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:50:17.139400 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:17.137440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:42051 (local address 127.3.0.254:39385)
0928 21:50:17.137490 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:17.137491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:17.137724 (+ 233us) server_negotiation.cc:407] Connection header received
0928 21:50:17.138147 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:17.138148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:17.138154 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:17.138206 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:17.139204 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:17.139205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:17.139205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:50:17.139208 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:17.139222 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:50:17.139225 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:17.139280 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:17.139339 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 21:50:17.139350 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:50:17.139570 13030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:17.137434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:39385 (local address 127.3.0.193:42051)
0928 21:50:17.137602 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:17.137633 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:17.138071 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:17.138263 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:17.138266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:17.138274 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:17.139066 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:50:17.139066 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:17.139093 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:50:17.139120 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:17.139287 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:17.139290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:17.139292 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:50:17.139339 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:50:17.139386 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260928 21:50:17.139998 13028 heartbeater.cc:347] Connected to a master server at 127.3.0.254:39385
I20260928 21:50:17.140079 13028 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:17.140200 13028 heartbeater.cc:511] Master 127.3.0.254:39385 requested a full tablet report, sending...
I20260928 21:50:17.140763 12840 ts_manager.cc:194] Registered new tserver with Master: e659ea6abf1641fe8c54fc70d3f68f45 (127.3.0.193:45399)
I20260928 21:50:17.140769 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 12900
I20260928 21:50:17.140844 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:50:17.141052 3075 external_mini_cluster.cc:1428] Setting key e49ac37027c4dfeec7fb07624ea736db
I20260928 21:50:17.141822 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39385
--builtin_ntp_servers=127.3.0.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:50:17.219417 13031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:17.219619 13031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:17.219638 13031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:17.220968 13031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:17.221020 13031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:17.221046 13031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:50:17.222513 13031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:17.222744 13031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:17.223001 13031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:17.227663 13037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:17.227669 13036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:17.227679 13039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:17.227788 13031 server_base.cc:1034] running on GCE node
I20260928 21:50:17.228085 13031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:17.228293 13031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:17.229475 13031 hybrid_clock.cc:648] HybridClock initialized: now 1790632217229442 us; error 52 us; skew 500 ppm
I20260928 21:50:17.230682 13031 webserver.cc:460] Webserver started at http://127.3.0.194:40197/ using document root <none> and password file <none>
I20260928 21:50:17.230894 13031 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:17.230934 13031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:17.231027 13031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:17.232059 13031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "69ac48cede0a4024adf6a446a05464f3"
format_stamp: "Formatted at 2026-09-28 21:50:17 on dist-test-slave-jrck"
server_key: "2a7c8315b60a3a340860dfe86c9bd486"
server_key_iv: "8dfff5e3f45ce256bca0b3b7bce6b9b4"
server_key_version: "encryptionkey@0"
I20260928 21:50:17.232352 13031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "69ac48cede0a4024adf6a446a05464f3"
format_stamp: "Formatted at 2026-09-28 21:50:17 on dist-test-slave-jrck"
server_key: "2a7c8315b60a3a340860dfe86c9bd486"
server_key_iv: "8dfff5e3f45ce256bca0b3b7bce6b9b4"
server_key_version: "encryptionkey@0"
I20260928 21:50:17.233547 13031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:50:17.234167 13045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:17.234301 13031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:50:17.234380 13031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "69ac48cede0a4024adf6a446a05464f3"
format_stamp: "Formatted at 2026-09-28 21:50:17 on dist-test-slave-jrck"
server_key: "2a7c8315b60a3a340860dfe86c9bd486"
server_key_iv: "8dfff5e3f45ce256bca0b3b7bce6b9b4"
server_key_version: "encryptionkey@0"
I20260928 21:50:17.234437 13031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:17.237982 13031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:17.238221 13031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:17.238333 13031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:17.238511 13031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:17.238797 13031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:17.238829 13031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:17.238854 13031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:17.238871 13031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:17.244150 13031 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:37395
I20260928 21:50:17.244204 13158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:37395 every 8 connection(s)
I20260928 21:50:17.244524 13031 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:50:17.245772 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 13031
I20260928 21:50:17.245867 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:50:17.246074 3075 external_mini_cluster.cc:1428] Setting key 0056a93f9c20101e224af5c246b1feac
I20260928 21:50:17.247020 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39385
--builtin_ntp_servers=127.3.0.212:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 21:50:17.247582 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:17.245194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:57863 (local address 127.3.0.254:39385)
0928 21:50:17.245251 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:17.245253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:17.245455 (+ 202us) server_negotiation.cc:407] Connection header received
0928 21:50:17.246052 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:17.246053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:17.246059 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:17.246104 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:17.247380 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:17.247381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:17.247382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:50:17.247385 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:17.247402 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:50:17.247405 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:17.247448 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:17.247519 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 21:50:17.247529 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:50:17.247797 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:17.245178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:39385 (local address 127.3.0.194:57863)
0928 21:50:17.245344 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:17.245377 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:17.245943 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:17.246192 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:17.246196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:17.246206 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:17.247229 (+ 1023us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:50:17.247230 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:17.247267 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:50:17.247301 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:17.247451 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:17.247455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:17.247458 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:50:17.247530 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:50:17.247594 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260928 21:50:17.248168 13159 heartbeater.cc:347] Connected to a master server at 127.3.0.254:39385
I20260928 21:50:17.248256 13159 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:17.248390 13159 heartbeater.cc:511] Master 127.3.0.254:39385 requested a full tablet report, sending...
I20260928 21:50:17.248768 12840 ts_manager.cc:194] Registered new tserver with Master: 69ac48cede0a4024adf6a446a05464f3 (127.3.0.194:37395)
W20260928 21:50:17.325511 13162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:17.325716 13162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:17.325743 13162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:17.326910 13162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:17.326948 13162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:17.326982 13162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:50:17.328294 13162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:17.328559 13162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:17.328827 13162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:17.333664 13167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:17.333813 13162 server_base.cc:1034] running on GCE node
W20260928 21:50:17.333693 13170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:17.333669 13168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:17.334097 13162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:17.334331 13162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:17.335470 13162 hybrid_clock.cc:648] HybridClock initialized: now 1790632217335464 us; error 33 us; skew 500 ppm
I20260928 21:50:17.336695 13162 webserver.cc:460] Webserver started at http://127.3.0.195:41405/ using document root <none> and password file <none>
I20260928 21:50:17.336911 13162 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:17.336956 13162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:17.337071 13162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:17.338150 13162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "8ccbebc83c2b4c2da7351c9162bde140"
format_stamp: "Formatted at 2026-09-28 21:50:17 on dist-test-slave-jrck"
server_key: "928afe8c53a5f32bf9958bef90612da3"
server_key_iv: "7afd2f6be40767658bd92fde5cb11e7f"
server_key_version: "encryptionkey@0"
I20260928 21:50:17.338477 13162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "8ccbebc83c2b4c2da7351c9162bde140"
format_stamp: "Formatted at 2026-09-28 21:50:17 on dist-test-slave-jrck"
server_key: "928afe8c53a5f32bf9958bef90612da3"
server_key_iv: "7afd2f6be40767658bd92fde5cb11e7f"
server_key_version: "encryptionkey@0"
I20260928 21:50:17.339601 13162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:50:17.340193 13176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:17.340346 13162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:17.340422 13162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "8ccbebc83c2b4c2da7351c9162bde140"
format_stamp: "Formatted at 2026-09-28 21:50:17 on dist-test-slave-jrck"
server_key: "928afe8c53a5f32bf9958bef90612da3"
server_key_iv: "7afd2f6be40767658bd92fde5cb11e7f"
server_key_version: "encryptionkey@0"
I20260928 21:50:17.340512 13162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:17.343677 13162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:17.343924 13162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:17.344040 13162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:17.344228 13162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:17.344554 13162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:17.344589 13162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:17.344615 13162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:17.344638 13162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:17.350137 13162 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:46249
I20260928 21:50:17.350173 13289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:46249 every 8 connection(s)
I20260928 21:50:17.350486 13162 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:50:17.351605 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 13162
I20260928 21:50:17.351688 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:50:17.351871 3075 external_mini_cluster.cc:1428] Setting key b8a0d4a6798fd901d3bfa1c5ba4b0789
I20260928 21:50:17.353417 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:17.351172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:49843 (local address 127.3.0.254:39385)
0928 21:50:17.351229 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:17.351232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:17.351442 (+ 210us) server_negotiation.cc:407] Connection header received
0928 21:50:17.351932 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:17.351933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:17.351941 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:17.352021 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:17.353171 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:17.353172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:17.353173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:50:17.353176 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:17.353191 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:50:17.353195 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:17.353236 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:17.353349 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 21:50:17.353366 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:50:17.353502 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:17.351151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:39385 (local address 127.3.0.195:49843)
0928 21:50:17.351324 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:17.351365 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:17.351838 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:17.352097 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:17.352101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:17.352111 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:17.353043 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:50:17.353043 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:17.353072 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:50:17.353100 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:17.353237 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:17.353239 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:17.353242 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:50:17.353296 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:50:17.353341 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260928 21:50:17.353915 13290 heartbeater.cc:347] Connected to a master server at 127.3.0.254:39385
I20260928 21:50:17.353997 13290 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:17.354105 13290 heartbeater.cc:511] Master 127.3.0.254:39385 requested a full tablet report, sending...
I20260928 21:50:17.354411 12840 ts_manager.cc:194] Registered new tserver with Master: 8ccbebc83c2b4c2da7351c9162bde140 (127.3.0.195:46249)
I20260928 21:50:17.363550 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:50:17.365293 12880 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:50:17.364848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.3.0.254:39385)
0928 21:50:17.364921 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:17.364923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:17.364990 (+ 67us) server_negotiation.cc:407] Connection header received
0928 21:50:17.365045 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:17.365046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:17.365051 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:17.365084 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:17.365221 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37894: BlockingRecv error: recv got EOF from 127.0.0.1:37894 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:50:17.365353 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 12900
I20260928 21:50:17.369232 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 13031
I20260928 21:50:17.372946 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 13162
I20260928 21:50:17.376363 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 12809
2026-09-28T21:50:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (546 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:50:17 dist-test-slave-jrck krb5kdc[13300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:50:17 dist-test-slave-jrck krb5kdc[13300](info): set up 2 sockets
Sep 28 21:50:17 dist-test-slave-jrck krb5kdc[13300](info): commencing operation
krb5kdc: starting...
W20260928 21:50:18.324533 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:50:18 dist-test-slave-jrck krb5kdc[13300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:50:18.432987 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:34997
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:34997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:50:18.510629 13313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:18.510828 13313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:18.510849 13313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:18.511935 13313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:50:18.511967 13313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:18.511981 13313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:18.511991 13313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:50:18.512001 13313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:50:18.513394 13313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:34997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:34997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:18.513579 13313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:18.513738 13313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:50:18.518227 13313 server_base.cc:1034] running on GCE node
W20260928 21:50:18.518195 13319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:18.518216 13321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:18.518189 13318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:18.518622 13313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:18.518886 13313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:18.520023 13313 hybrid_clock.cc:648] HybridClock initialized: now 1790632218520010 us; error 29 us; skew 500 ppm
Sep 28 21:50:18 dist-test-slave-jrck krb5kdc[13300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632218, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:18.522468 13313 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:50:18.522928 13313 webserver.cc:460] Webserver started at http://127.3.0.254:38761/ using document root <none> and password file <none>
I20260928 21:50:18.523135 13313 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:18.523185 13313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:18.523311 13313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:18.524322 13313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "4099f8ea26ad478fba216596d445cd21"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "7eb6f631e6af859270bbce63a180d9d4"
server_key_iv: "ad565504af8123d56144a6968df72215"
server_key_version: "encryptionkey@0"
I20260928 21:50:18.524669 13313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "4099f8ea26ad478fba216596d445cd21"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "7eb6f631e6af859270bbce63a180d9d4"
server_key_iv: "ad565504af8123d56144a6968df72215"
server_key_version: "encryptionkey@0"
I20260928 21:50:18.525888 13313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:50:18.526527 13328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:18.526669 13313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:18.526757 13313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "4099f8ea26ad478fba216596d445cd21"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "7eb6f631e6af859270bbce63a180d9d4"
server_key_iv: "ad565504af8123d56144a6968df72215"
server_key_version: "encryptionkey@0"
I20260928 21:50:18.526825 13313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:18.537789 13313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:18.539523 13313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:18.539641 13313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:18.543231 13313 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:34997
I20260928 21:50:18.543256 13380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:34997 every 8 connection(s)
I20260928 21:50:18.543567 13313 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:50:18.544540 13381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:50:18.546710 13381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21: Bootstrap starting.
I20260928 21:50:18.546941 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 13313
I20260928 21:50:18.547043 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:50:18.547226 3075 external_mini_cluster.cc:1428] Setting key 549cdc1bcc85afb85a91e4498baaf3fe
I20260928 21:50:18.547399 13381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21: Neither blocks nor log segments found. Creating new log.
I20260928 21:50:18.547693 13381 log.cc:824] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21: Log is configured to *not* fsync() on all Append() calls
I20260928 21:50:18.548345 13381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21: No bootstrap required, opened a new log
I20260928 21:50:18.550400 13381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4099f8ea26ad478fba216596d445cd21" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34997 } }
I20260928 21:50:18.550575 13381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:50:18.550611 13381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4099f8ea26ad478fba216596d445cd21, State: Initialized, Role: FOLLOWER
I20260928 21:50:18.550734 13381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4099f8ea26ad478fba216596d445cd21" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34997 } }
I20260928 21:50:18.550828 13381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:50:18.550863 13381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:50:18.550906 13381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:50:18.551467 13381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4099f8ea26ad478fba216596d445cd21" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34997 } }
I20260928 21:50:18.551555 13381 leader_election.cc:302] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4099f8ea26ad478fba216596d445cd21; no voters:
I20260928 21:50:18.551715 13381 leader_election.cc:288] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:50:18.551748 13386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:50:18.551944 13386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [term 1 LEADER]: Becoming Leader. State: Replica: 4099f8ea26ad478fba216596d445cd21, State: Running, Role: LEADER
I20260928 21:50:18.552026 13381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:50:18.552073 13386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4099f8ea26ad478fba216596d445cd21" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34997 } }
I20260928 21:50:18.552487 13387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4099f8ea26ad478fba216596d445cd21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4099f8ea26ad478fba216596d445cd21" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34997 } } }
I20260928 21:50:18.552578 13387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [sys.catalog]: This master's current role is: LEADER
I20260928 21:50:18.552495 13388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4099f8ea26ad478fba216596d445cd21. Latest consensus state: current_term: 1 leader_uuid: "4099f8ea26ad478fba216596d445cd21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4099f8ea26ad478fba216596d445cd21" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 34997 } } }
I20260928 21:50:18.552759 13388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21 [sys.catalog]: This master's current role is: LEADER
Sep 28 21:50:18 dist-test-slave-jrck krb5kdc[13300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
W20260928 21:50:18.553860 13401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:50:18.553905 13401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:50:18.553972 13394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:50:18.554107 13394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:50:18.556743 13394 catalog_manager.cc:1347] Generated new cluster ID: 878bd41e7e1c4c849037baa50019e17f
I20260928 21:50:18.556787 13394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:50:18.557610 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:18.547729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.3.0.254:34997)
0928 21:50:18.547908 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:18.547917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:18.547945 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:50:18.548360 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:18.548366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:18.548425 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:18.548588 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:18.549278 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:18.550159 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:18.551167 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:18.551599 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:18.551708 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:18.554366 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:18.554370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:18.554370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:50:18.554383 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:18.555929 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:18.556314 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:18.556318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:18.556318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:18.556439 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:18.556686 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:18.556686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:18.556687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:18.556988 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:18.557147 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:18.557252 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 21:50:18.557400 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260928 21:50:18.579365 13394 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:50:18.579891 13394 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:50:18.582391 13394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21: Generated new TSK 0
I20260928 21:50:18.582566 13394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:50:18.610868 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:34997
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:50:18.689097 13407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:18.689296 13407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:18.689327 13407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:18.690483 13407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:18.690526 13407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:18.690559 13407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:50:18.691977 13407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:34997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:18.692190 13407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:18.692389 13407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:18.697062 13415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:18.697180 13407 server_base.cc:1034] running on GCE node
W20260928 21:50:18.697062 13412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:18.697069 13413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:18.697458 13407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:18.697685 13407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:18.698832 13407 hybrid_clock.cc:648] HybridClock initialized: now 1790632218698818 us; error 29 us; skew 500 ppm
Sep 28 21:50:18 dist-test-slave-jrck krb5kdc[13300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632218, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:18.701126 13407 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:50:18.701506 13407 webserver.cc:460] Webserver started at http://127.3.0.193:34849/ using document root <none> and password file <none>
I20260928 21:50:18.701690 13407 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:18.701735 13407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:18.701881 13407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:18.702870 13407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "03f239a1b74a4b45919ca8aec674b88b"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "32e03de4b3be52249c955fa22def78eb"
server_key_iv: "1f9c2ed58f566c99ced3621f614aca76"
server_key_version: "encryptionkey@0"
I20260928 21:50:18.703186 13407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "03f239a1b74a4b45919ca8aec674b88b"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "32e03de4b3be52249c955fa22def78eb"
server_key_iv: "1f9c2ed58f566c99ced3621f614aca76"
server_key_version: "encryptionkey@0"
I20260928 21:50:18.704330 13407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:50:18.705014 13422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:18.705161 13407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:18.705240 13407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "03f239a1b74a4b45919ca8aec674b88b"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "32e03de4b3be52249c955fa22def78eb"
server_key_iv: "1f9c2ed58f566c99ced3621f614aca76"
server_key_version: "encryptionkey@0"
I20260928 21:50:18.705303 13407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:18.734726 13407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:18.736522 13407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:18.736660 13407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:18.736896 13407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:18.737185 13407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:18.737217 13407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:18.737246 13407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:18.737268 13407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:18.742811 13407 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:44913
I20260928 21:50:18.742882 13535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:44913 every 8 connection(s)
I20260928 21:50:18.743165 13407 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:50:18.745504 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 13407
I20260928 21:50:18.745592 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:50:18.745775 3075 external_mini_cluster.cc:1428] Setting key 18ca17ce9994780eb6bf758807c552c1
Sep 28 21:50:18 dist-test-slave-jrck krb5kdc[13300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632218, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:50:18.752931 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:18.743850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:46237 (local address 127.3.0.254:34997)
0928 21:50:18.743912 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:18.743914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:18.744102 (+ 188us) server_negotiation.cc:407] Connection header received
0928 21:50:18.744606 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:18.744608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:18.744617 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:18.744668 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:18.745613 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:18.746297 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:18.747697 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:18.747998 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:18.748068 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:18.750394 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:18.750396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:18.750396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:50:18.750403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:18.751657 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:18.752011 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:18.752012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:18.752012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:18.752063 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:18.752272 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:18.752273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:18.752273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:18.752415 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:18.752499 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:18.752750 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 21:50:18.752831 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:50:18.753083 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:18.743832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:34997 (local address 127.3.0.193:46237)
0928 21:50:18.743985 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:18.744023 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:18.744523 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:18.744725 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:18.744729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:18.745291 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:18.745513 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:18.745518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:18.746388 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:18.746389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:18.747608 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:18.747611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:18.748076 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:18.748077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:18.748178 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:18.748703 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:18.748711 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:18.750305 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:18.751727 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:18.751734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:18.751736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:18.751948 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:18.752116 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:18.752117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:18.752117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:18.752220 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:18.752487 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:18.752494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:18.752635 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 21:50:18.752722 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:50:18.752872 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
I20260928 21:50:18.753732 13536 heartbeater.cc:347] Connected to a master server at 127.3.0.254:34997
I20260928 21:50:18.753822 13536 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:18.754004 13536 heartbeater.cc:511] Master 127.3.0.254:34997 requested a full tablet report, sending...
I20260928 21:50:18.754546 13345 ts_manager.cc:194] Registered new tserver with Master: 03f239a1b74a4b45919ca8aec674b88b (127.3.0.193:44913)
I20260928 21:50:18.755137 13345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:46237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:50:18.767772 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:34997
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:50:18.846928 13541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:18.847112 13541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:18.847133 13541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:18.848224 13541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:18.848263 13541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:18.848305 13541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:50:18.849977 13541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:34997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:18.850207 13541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:18.850457 13541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:18.855533 13546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:18.855567 13549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:18.855546 13547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:18.855676 13541 server_base.cc:1034] running on GCE node
I20260928 21:50:18.855994 13541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:18.856233 13541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:18.857370 13541 hybrid_clock.cc:648] HybridClock initialized: now 1790632218857356 us; error 28 us; skew 500 ppm
Sep 28 21:50:18 dist-test-slave-jrck krb5kdc[13300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632218, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:18.859676 13541 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:50:18.860051 13541 webserver.cc:460] Webserver started at http://127.3.0.194:36779/ using document root <none> and password file <none>
I20260928 21:50:18.860219 13541 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:18.860255 13541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:18.860361 13541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:18.861374 13541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "d7cceffbd5484eec9ab6af7053208d3b"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "cd8c717f529e85564d64b9a129de0ecf"
server_key_iv: "e1a042f0a3dfea5cb73856dce7ec62c7"
server_key_version: "encryptionkey@0"
I20260928 21:50:18.861651 13541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "d7cceffbd5484eec9ab6af7053208d3b"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "cd8c717f529e85564d64b9a129de0ecf"
server_key_iv: "e1a042f0a3dfea5cb73856dce7ec62c7"
server_key_version: "encryptionkey@0"
I20260928 21:50:18.862753 13541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:50:18.863359 13556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:18.863494 13541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:18.863569 13541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "d7cceffbd5484eec9ab6af7053208d3b"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "cd8c717f529e85564d64b9a129de0ecf"
server_key_iv: "e1a042f0a3dfea5cb73856dce7ec62c7"
server_key_version: "encryptionkey@0"
I20260928 21:50:18.863618 13541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:18.875209 13541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:18.876850 13541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:18.876951 13541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:18.877146 13541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:18.877432 13541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:18.877465 13541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:18.877486 13541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:18.877508 13541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:18.882808 13541 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:38073
I20260928 21:50:18.882845 13669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:38073 every 8 connection(s)
I20260928 21:50:18.883114 13541 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:50:18 dist-test-slave-jrck krb5kdc[13300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632218, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:50:18.891042 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:18.883741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:41595 (local address 127.3.0.254:34997)
0928 21:50:18.883793 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:18.883795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:18.883993 (+ 198us) server_negotiation.cc:407] Connection header received
0928 21:50:18.884478 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:18.884479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:18.884490 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:18.884550 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:18.885396 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:18.885875 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:18.886755 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:18.886940 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:18.886983 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:18.888889 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:18.888890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:18.888890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:50:18.888897 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:18.890073 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:18.890378 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:18.890379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:18.890379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:18.890425 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:18.890593 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:18.890594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:18.890594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:18.890684 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:18.890726 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:18.890922 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 21:50:18.890979 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:50:18.891215 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:18.883707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:34997 (local address 127.3.0.194:41595)
0928 21:50:18.883876 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:18.883916 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:18.884379 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:18.884616 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:18.884620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:18.885150 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:18.885332 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:18.885337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:18.885932 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:18.885933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:18.886698 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:18.886700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:18.886995 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:18.886995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:18.887089 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:18.887497 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:18.887504 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:18.888795 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:18.890132 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:18.890135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:18.890136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:18.890317 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:18.890468 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:18.890468 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:18.890468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:18.890552 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:18.890724 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:18.890726 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:18.890831 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 21:50:18.890906 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:50:18.891017 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260928 21:50:18.891636 13670 heartbeater.cc:347] Connected to a master server at 127.3.0.254:34997
I20260928 21:50:18.891714 13670 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:18.891884 13670 heartbeater.cc:511] Master 127.3.0.254:34997 requested a full tablet report, sending...
I20260928 21:50:18.891920 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 13541
I20260928 21:50:18.892009 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:50:18.892216 3075 external_mini_cluster.cc:1428] Setting key e7a65b5578b4af7c674e938b03f424e5
I20260928 21:50:18.892235 13345 ts_manager.cc:194] Registered new tserver with Master: d7cceffbd5484eec9ab6af7053208d3b (127.3.0.194:38073)
I20260928 21:50:18.892704 13345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:41595
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:50:18.912782 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:34997
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:50:18.991016 13675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:50:18.991225 13675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:50:18.991246 13675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:50:18.992434 13675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:50:18.992501 13675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:50:18.992527 13675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:50:18.993990 13675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:34997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:50:18.994177 13675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:50:18.994354 13675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:50:18.998901 13683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:18.998903 13680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:50:18.998961 13681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:50:18.999213 13675 server_base.cc:1034] running on GCE node
I20260928 21:50:18.999432 13675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:50:18.999648 13675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:50:19.000790 13675 hybrid_clock.cc:648] HybridClock initialized: now 1790632219000777 us; error 29 us; skew 500 ppm
Sep 28 21:50:18 dist-test-slave-jrck krb5kdc[13300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632218, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:50:19.003093 13675 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:50:19.003508 13675 webserver.cc:460] Webserver started at http://127.3.0.195:41629/ using document root <none> and password file <none>
I20260928 21:50:19.003706 13675 fs_manager.cc:359] Metadata directory not provided
I20260928 21:50:19.003753 13675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:50:19.003885 13675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:50:19.004930 13675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "24fd78c7862341eca82a59736ccea5dc"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "f3d2d6855d6a514ad03c12bedabc3e32"
server_key_iv: "b740c5106e6b2a7735241ce470c107af"
server_key_version: "encryptionkey@0"
I20260928 21:50:19.005244 13675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "24fd78c7862341eca82a59736ccea5dc"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "f3d2d6855d6a514ad03c12bedabc3e32"
server_key_iv: "b740c5106e6b2a7735241ce470c107af"
server_key_version: "encryptionkey@0"
I20260928 21:50:19.006426 13675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:50:19.007043 13690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:19.007189 13675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:50:19.007266 13675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "24fd78c7862341eca82a59736ccea5dc"
format_stamp: "Formatted at 2026-09-28 21:50:18 on dist-test-slave-jrck"
server_key: "f3d2d6855d6a514ad03c12bedabc3e32"
server_key_iv: "b740c5106e6b2a7735241ce470c107af"
server_key_version: "encryptionkey@0"
I20260928 21:50:19.007336 13675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:50:19.019470 13675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:50:19.021533 13675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:50:19.021664 13675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:50:19.021857 13675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:50:19.022133 13675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:50:19.022166 13675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:19.022192 13675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:50:19.022210 13675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:50:19.027719 13675 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:45935
I20260928 21:50:19.027767 13803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:45935 every 8 connection(s)
I20260928 21:50:19.028043 13675 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:50:19 dist-test-slave-jrck krb5kdc[13300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632218, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:50:19.036340 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.028719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:37749 (local address 127.3.0.254:34997)
0928 21:50:19.028774 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:19.028776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:19.028976 (+ 200us) server_negotiation.cc:407] Connection header received
0928 21:50:19.029474 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:19.029475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:19.029487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:19.029537 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:19.030473 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.030958 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.031910 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.032105 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.032157 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.034044 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:19.034046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:19.034046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:50:19.034053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:19.035109 (+ 1056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:19.035460 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:19.035461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:19.035461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:19.035512 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:19.035715 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:19.035716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:19.035717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:19.035827 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:19.035882 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:19.036132 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 21:50:19.036225 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:50:19.036491 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.028713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:34997 (local address 127.3.0.195:37749)
0928 21:50:19.028863 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:19.028897 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:19.029382 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:19.029606 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:19.029610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:19.030198 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:50:19.030381 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.030386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.031036 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.031037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.031829 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.031831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.032167 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.032168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.032290 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.032684 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:50:19.032692 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:50:19.033959 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:50:19.035180 (+ 1221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:19.035187 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:19.035189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:19.035394 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:19.035564 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:50:19.035565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:50:19.035565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:50:19.035661 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:50:19.035880 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:50:19.035886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:50:19.036017 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 21:50:19.036104 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:50:19.036254 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260928 21:50:19.036907 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 13675
I20260928 21:50:19.036967 13804 heartbeater.cc:347] Connected to a master server at 127.3.0.254:34997
I20260928 21:50:19.037004 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:50:19.037070 13804 heartbeater.cc:464] Registering TS with master...
I20260928 21:50:19.037191 3075 external_mini_cluster.cc:1428] Setting key d9f8fcaf77407b60fa163894f0961418
I20260928 21:50:19.037240 13804 heartbeater.cc:511] Master 127.3.0.254:34997 requested a full tablet report, sending...
I20260928 21:50:19.037673 13344 ts_manager.cc:194] Registered new tserver with Master: 24fd78c7862341eca82a59736ccea5dc (127.3.0.195:45935)
I20260928 21:50:19.038022 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:50:19.038115 13344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:37749
I20260928 21:50:19.044445 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.039260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.3.0.254:34997)
0928 21:50:19.039300 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:19.039302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:19.039409 (+ 107us) server_negotiation.cc:407] Connection header received
0928 21:50:19.039466 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:19.039467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:19.039476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:19.039525 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:50:19.040049 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.040559 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.041287 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.041477 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.041529 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.042507 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:50:19.042508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:50:19.042509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:50:19.042515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:50:19.043469 (+ 954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:19.043752 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:19.043753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:19.043754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:19.043805 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:50:19.043961 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:50:19.043962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:50:19.043962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:50:19.044095 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:50:19.044183 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:19.044291 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 21:50:19.044359 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:50:19.045693 13344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:50:19.047132 13344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:50:19.053023 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.049395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.3.0.195:45935)
0928 21:50:19.049549 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:19.049552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:19.049560 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:50:19.049592 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:19.049594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:19.049609 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:19.049898 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:50:19.049960 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.050998 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.052017 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.052671 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.052737 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.052889 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:19.052913 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 21:50:19.052948 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260928 21:50:19.053012 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.049222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:45935 (local address 127.0.0.1:48306)
0928 21:50:19.049384 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:19.049392 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:19.049425 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:19.049694 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:19.049694 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:19.049696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:50:19.049760 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.049762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.051111 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.051112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.051908 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.051911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.052740 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.052741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.052817 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.052830 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 21:50:19.052889 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:50:19.052939 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":7,"thread_start_us":44,"threads_started":1}
I20260928 21:50:19.053289 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.049045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:44913 (local address 127.0.0.1:51730)
0928 21:50:19.049167 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:19.049206 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:19.049264 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:19.049684 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:19.049685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:19.049687 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:50:19.049753 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.049755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.051079 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.051080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.052126 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.052128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.053096 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.053097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.053148 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.053151 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:50:19.053176 (+ 25us) client_negotiation.cc:240] Negotiation successful
0928 21:50:19.053208 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":4,"thread_start_us":50,"threads_started":1}
I20260928 21:50:19.053292 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.049057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:38073 (local address 127.0.0.1:41090)
0928 21:50:19.049322 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:19.049332 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:19.049386 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:19.049662 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:19.049665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:19.049667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:50:19.049752 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.049754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.050684 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.050685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.052075 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.052078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.053010 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.053011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.053105 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.053109 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:50:19.053138 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 21:50:19.053220 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":22,"thread_start_us":60,"threads_started":1}
I20260928 21:50:19.053548 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.049207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.3.0.194:38073)
0928 21:50:19.049497 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:19.049502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:19.049511 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:50:19.049549 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:19.049551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:19.049566 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:19.049638 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:50:19.049817 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.050589 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.052198 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.052913 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.053327 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.053405 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:19.053431 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 21:50:19.053469 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":79,"threads_started":1}
I20260928 21:50:19.053643 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.049154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.3.0.193:44913)
0928 21:50:19.049522 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:19.049526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:19.049535 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:50:19.049572 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:19.049574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:19.049588 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:19.049824 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:50:19.049894 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.050979 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.052204 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.053017 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.053463 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.053518 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:19.053539 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 21:50:19.053575 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":77,"threads_started":1}
I20260928 21:50:19.054137 13604 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbf07aabd2b4e8abafc838489495033 (DEFAULT_TABLE table=test-table [id=3fb56a4abfa84e5fbc9474a687bcf5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:50:19.054174 13470 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbf07aabd2b4e8abafc838489495033 (DEFAULT_TABLE table=test-table [id=3fb56a4abfa84e5fbc9474a687bcf5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:50:19.054163 13738 tablet_service.cc:1467] Processing CreateTablet for tablet 3bbf07aabd2b4e8abafc838489495033 (DEFAULT_TABLE table=test-table [id=3fb56a4abfa84e5fbc9474a687bcf5f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:50:19.054458 13604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bbf07aabd2b4e8abafc838489495033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:50:19.054466 13738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bbf07aabd2b4e8abafc838489495033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:50:19.054466 13470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bbf07aabd2b4e8abafc838489495033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:50:19.056824 13818 tablet_bootstrap.cc:492] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: Bootstrap starting.
I20260928 21:50:19.057435 13818 tablet_bootstrap.cc:654] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: Neither blocks nor log segments found. Creating new log.
I20260928 21:50:19.057507 13819 tablet_bootstrap.cc:492] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: Bootstrap starting.
I20260928 21:50:19.057905 13820 tablet_bootstrap.cc:492] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b: Bootstrap starting.
I20260928 21:50:19.058128 13819 tablet_bootstrap.cc:654] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: Neither blocks nor log segments found. Creating new log.
I20260928 21:50:19.058292 13818 log.cc:824] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:50:19.058485 13820 tablet_bootstrap.cc:654] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b: Neither blocks nor log segments found. Creating new log.
I20260928 21:50:19.058647 13819 log.cc:824] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:50:19.058979 13820 log.cc:824] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:50:19.059084 13818 tablet_bootstrap.cc:492] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: No bootstrap required, opened a new log
I20260928 21:50:19.059149 13818 ts_tablet_manager.cc:1397] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:50:19.059337 13819 tablet_bootstrap.cc:492] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: No bootstrap required, opened a new log
I20260928 21:50:19.059402 13819 ts_tablet_manager.cc:1397] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:50:19.059633 13820 tablet_bootstrap.cc:492] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b: No bootstrap required, opened a new log
I20260928 21:50:19.059688 13820 ts_tablet_manager.cc:1397] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:50:19.061076 13818 raft_consensus.cc:348] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f239a1b74a4b45919ca8aec674b88b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44913 } } peers { permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 } } peers { permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 } }
I20260928 21:50:19.061249 13818 raft_consensus.cc:374] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:50:19.061177 13820 raft_consensus.cc:348] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f239a1b74a4b45919ca8aec674b88b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44913 } } peers { permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 } } peers { permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 } }
I20260928 21:50:19.061283 13818 raft_consensus.cc:729] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cceffbd5484eec9ab6af7053208d3b, State: Initialized, Role: FOLLOWER
I20260928 21:50:19.061300 13820 raft_consensus.cc:374] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:50:19.061209 13819 raft_consensus.cc:348] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f239a1b74a4b45919ca8aec674b88b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44913 } } peers { permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 } } peers { permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 } }
I20260928 21:50:19.061331 13820 raft_consensus.cc:729] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f239a1b74a4b45919ca8aec674b88b, State: Initialized, Role: FOLLOWER
I20260928 21:50:19.061342 13819 raft_consensus.cc:374] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:50:19.061372 13819 raft_consensus.cc:729] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fd78c7862341eca82a59736ccea5dc, State: Initialized, Role: FOLLOWER
I20260928 21:50:19.061447 13820 consensus_queue.cc:260] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f239a1b74a4b45919ca8aec674b88b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44913 } } peers { permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 } } peers { permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 } }
I20260928 21:50:19.061489 13819 consensus_queue.cc:260] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f239a1b74a4b45919ca8aec674b88b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44913 } } peers { permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 } } peers { permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 } }
I20260928 21:50:19.061480 13818 consensus_queue.cc:260] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f239a1b74a4b45919ca8aec674b88b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44913 } } peers { permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 } } peers { permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 } }
I20260928 21:50:19.061709 13820 ts_tablet_manager.cc:1428] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:50:19.061729 13819 ts_tablet_manager.cc:1428] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:50:19.061729 13818 ts_tablet_manager.cc:1428] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:50:19.061758 13804 heartbeater.cc:503] Master 127.3.0.254:34997 was elected leader, sending a full tablet report...
I20260928 21:50:19.061810 13536 heartbeater.cc:503] Master 127.3.0.254:34997 was elected leader, sending a full tablet report...
I20260928 21:50:19.061846 13670 heartbeater.cc:503] Master 127.3.0.254:34997 was elected leader, sending a full tablet report...
W20260928 21:50:19.133622 13671 tablet.cc:2369] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:50:19.243872 13537 tablet.cc:2369] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:50:19.278548 13805 tablet.cc:2369] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:50:19.405349 13824 raft_consensus.cc:482] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:50:19.405467 13824 raft_consensus.cc:504] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f239a1b74a4b45919ca8aec674b88b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44913 } } peers { permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 } } peers { permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 } }
I20260928 21:50:19.405747 13824 leader_election.cc:288] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cceffbd5484eec9ab6af7053208d3b (127.3.0.194:38073), 24fd78c7862341eca82a59736ccea5dc (127.3.0.195:45935)
I20260928 21:50:19.408705 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.405906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:38073 (local address 127.3.0.193:45039)
0928 21:50:19.406037 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:19.406049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:19.406112 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:19.406343 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:19.406344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:19.406346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:50:19.406413 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.406414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.407043 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.407044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.407952 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.407954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.408465 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.408466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.408533 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.408538 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:50:19.408584 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:50:19.408631 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":41,"thread_start_us":79,"threads_started":1}
I20260928 21:50:19.408711 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.405973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:45039 (local address 127.3.0.194:38073)
0928 21:50:19.406027 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:19.406029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:19.406106 (+ 77us) server_negotiation.cc:407] Connection header received
0928 21:50:19.406247 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:19.406248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:19.406259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:19.406284 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:50:19.406463 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.406964 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.408039 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.408399 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.408448 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.408477 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:19.408606 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 21:50:19.408642 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:50:19.408842 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.406157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:52983 (local address 127.3.0.195:45935)
0928 21:50:19.406215 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:19.406217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:19.406266 (+ 49us) server_negotiation.cc:407] Connection header received
0928 21:50:19.406331 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:19.406332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:19.406344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:19.406392 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:50:19.406549 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.406999 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.408004 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.408511 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.408578 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.408587 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:19.408724 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 21:50:19.408757 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:50:19.408818 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.406095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:45935 (local address 127.3.0.193:52983)
0928 21:50:19.406193 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:19.406203 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:50:19.406270 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:50:19.406399 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:50:19.406400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:50:19.406401 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:50:19.406484 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.406485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.407075 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.407076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.407931 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:50:19.407933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.408582 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.408583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:50:19.408656 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.408660 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:50:19.408711 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:50:19.408747 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260928 21:50:19.409194 13624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbf07aabd2b4e8abafc838489495033" candidate_uuid: "03f239a1b74a4b45919ca8aec674b88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cceffbd5484eec9ab6af7053208d3b" is_pre_election: true
I20260928 21:50:19.409202 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbf07aabd2b4e8abafc838489495033" candidate_uuid: "03f239a1b74a4b45919ca8aec674b88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd78c7862341eca82a59736ccea5dc" is_pre_election: true
I20260928 21:50:19.409355 13624 raft_consensus.cc:2454] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03f239a1b74a4b45919ca8aec674b88b in term 0.
I20260928 21:50:19.409396 13758 raft_consensus.cc:2454] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03f239a1b74a4b45919ca8aec674b88b in term 0.
I20260928 21:50:19.409564 13423 leader_election.cc:302] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f239a1b74a4b45919ca8aec674b88b, d7cceffbd5484eec9ab6af7053208d3b; no voters:
I20260928 21:50:19.409680 13824 raft_consensus.cc:2781] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:50:19.409742 13824 raft_consensus.cc:482] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:50:19.409771 13824 raft_consensus.cc:3037] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:50:19.410502 13824 raft_consensus.cc:504] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f239a1b74a4b45919ca8aec674b88b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44913 } } peers { permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 } } peers { permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 } }
I20260928 21:50:19.410641 13824 leader_election.cc:288] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [CANDIDATE]: Term 1 election: Requested vote from peers d7cceffbd5484eec9ab6af7053208d3b (127.3.0.194:38073), 24fd78c7862341eca82a59736ccea5dc (127.3.0.195:45935)
I20260928 21:50:19.410796 13624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbf07aabd2b4e8abafc838489495033" candidate_uuid: "03f239a1b74a4b45919ca8aec674b88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cceffbd5484eec9ab6af7053208d3b"
I20260928 21:50:19.410833 13758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bbf07aabd2b4e8abafc838489495033" candidate_uuid: "03f239a1b74a4b45919ca8aec674b88b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24fd78c7862341eca82a59736ccea5dc"
I20260928 21:50:19.410903 13758 raft_consensus.cc:3037] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:50:19.410880 13624 raft_consensus.cc:3037] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:50:19.411590 13758 raft_consensus.cc:2454] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03f239a1b74a4b45919ca8aec674b88b in term 1.
I20260928 21:50:19.411590 13624 raft_consensus.cc:2454] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03f239a1b74a4b45919ca8aec674b88b in term 1.
I20260928 21:50:19.411743 13423 leader_election.cc:302] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f239a1b74a4b45919ca8aec674b88b, d7cceffbd5484eec9ab6af7053208d3b; no voters:
I20260928 21:50:19.411823 13824 raft_consensus.cc:2781] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:50:19.411953 13824 raft_consensus.cc:686] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [term 1 LEADER]: Becoming Leader. State: Replica: 03f239a1b74a4b45919ca8aec674b88b, State: Running, Role: LEADER
I20260928 21:50:19.412036 13824 consensus_queue.cc:237] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f239a1b74a4b45919ca8aec674b88b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44913 } } peers { permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 } } peers { permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 } }
I20260928 21:50:19.412824 13345 catalog_manager.cc:5495] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b reported cstate change: term changed from 0 to 1, leader changed from <none> to 03f239a1b74a4b45919ca8aec674b88b (127.3.0.193). New cstate: current_term: 1 leader_uuid: "03f239a1b74a4b45919ca8aec674b88b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f239a1b74a4b45919ca8aec674b88b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 } health_report { overall_health: UNKNOWN } } }
I20260928 21:50:19.494542 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.492150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.3.0.193:44913)
0928 21:50:19.492205 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:50:19.492206 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:50:19.492214 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:50:19.492256 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:50:19.492257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:50:19.492281 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:50:19.492350 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:50:19.492519 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.493024 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.493756 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:50:19.493966 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:50:19.494020 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:50:19.494217 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:50:19.494325 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:50:19.494405 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:50:19.494442 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 21:50:19.494463 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:50:19.496286 13758 raft_consensus.cc:1261] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc [term 1 FOLLOWER]: Refusing update from remote peer 03f239a1b74a4b45919ca8aec674b88b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:50:19.496326 13624 raft_consensus.cc:1261] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b [term 1 FOLLOWER]: Refusing update from remote peer 03f239a1b74a4b45919ca8aec674b88b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:50:19.496642 13824 consensus_queue.cc:1035] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24fd78c7862341eca82a59736ccea5dc" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:50:19.496778 13829 consensus_queue.cc:1035] T 3bbf07aabd2b4e8abafc838489495033 P 03f239a1b74a4b45919ca8aec674b88b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cceffbd5484eec9ab6af7053208d3b" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 38073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:50:19.501543 13345 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:50:19.501627 13345 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:50:19.502291 13345 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4099f8ea26ad478fba216596d445cd21: Sending DeleteTablet for 3 replicas of tablet 3bbf07aabd2b4e8abafc838489495033
I20260928 21:50:19.502607 13470 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbf07aabd2b4e8abafc838489495033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:50:19 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:51730
I20260928 21:50:19.502651 13604 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbf07aabd2b4e8abafc838489495033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:50:19 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:41090
I20260928 21:50:19.502813 13738 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bbf07aabd2b4e8abafc838489495033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:50:19 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:48306
I20260928 21:50:19.502884 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 13407
I20260928 21:50:19.503407 13845 tablet_replica.cc:330] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: stopping tablet replica
I20260928 21:50:19.503492 13845 raft_consensus.cc:2229] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:50:19.503563 13845 raft_consensus.cc:2258] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:50:19.504484 13844 tablet_replica.cc:330] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: stopping tablet replica
I20260928 21:50:19.504583 13844 raft_consensus.cc:2229] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:50:19.504601 13845 ts_tablet_manager.cc:1905] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:50:19.504680 13844 raft_consensus.cc:2258] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:50:19.505065 13844 ts_tablet_manager.cc:1905] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:50:19.506019 13845 ts_tablet_manager.cc:1918] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:50:19.506088 13845 log.cc:1196] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/3bbf07aabd2b4e8abafc838489495033
I20260928 21:50:19.506307 13845 ts_tablet_manager.cc:1939] T 3bbf07aabd2b4e8abafc838489495033 P 24fd78c7862341eca82a59736ccea5dc: Deleting consensus metadata
I20260928 21:50:19.506624 13844 ts_tablet_manager.cc:1918] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:50:19.506692 13844 log.cc:1196] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/3bbf07aabd2b4e8abafc838489495033
I20260928 21:50:19.506668 13329 catalog_manager.cc:4841] TS 24fd78c7862341eca82a59736ccea5dc (127.3.0.195:45935): tablet 3bbf07aabd2b4e8abafc838489495033 (table test-table [id=3fb56a4abfa84e5fbc9474a687bcf5f5]) successfully deleted
I20260928 21:50:19.506958 13844 ts_tablet_manager.cc:1939] T 3bbf07aabd2b4e8abafc838489495033 P d7cceffbd5484eec9ab6af7053208d3b: Deleting consensus metadata
I20260928 21:50:19.507701 13329 catalog_manager.cc:4841] TS d7cceffbd5484eec9ab6af7053208d3b (127.3.0.194:38073): tablet 3bbf07aabd2b4e8abafc838489495033 (table test-table [id=3fb56a4abfa84e5fbc9474a687bcf5f5]) successfully deleted
W20260928 21:50:19.508987 13330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:44913 (error 108)
I20260928 21:50:19.509192 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 13541
I20260928 21:50:19.509704 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:50:19.509295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:44913 (local address 127.0.0.1:51748)
0928 21:50:19.509637 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 21:50:19.509671 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:44913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325}
W20260928 21:50:19.510383 13330 catalog_manager.cc:4584] TS 03f239a1b74a4b45919ca8aec674b88b (127.3.0.193:44913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3bbf07aabd2b4e8abafc838489495033: Network error: Client connection negotiation failed: client connection to 127.3.0.193:44913: connect: Connection refused (error 111)
I20260928 21:50:19.513800 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 13675
I20260928 21:50:19.517598 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 13313
2026-09-28T21:50:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2142 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:50:19 dist-test-slave-jrck krb5kdc[13849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:50:19 dist-test-slave-jrck krb5kdc[13849](info): set up 2 sockets
Sep 28 21:50:19 dist-test-slave-jrck krb5kdc[13849](info): commencing operation
krb5kdc: starting...
W20260928 21:50:20.468477 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:50:20 dist-test-slave-jrck krb5kdc[13849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:50:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.0.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.0.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/rangeradmin_127.3.0.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/rangeradmin_127.3.0.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.0.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.0.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/rangerlookup_127.3.0.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/rangerlookup_127.3.0.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.0.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.0.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/HTTP_127.3.0.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/HTTP_127.3.0.212@KRBTEST.COM.keytab.
I20260928 21:50:20.616165 3075 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksdiuc7/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/postgres -l logfile start
2026-09-28 21:50:22.793 UTC [13879] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:50:22.793 UTC [13879] LOG: listening on IPv4 address "127.3.0.212", port 59809
2026-09-28 21:50:22.796 UTC [13879] LOG: listening on Unix socket "/tmp/.s.PGSQL.59809"
2026-09-28 21:50:22.822 UTC [13882] LOG: database system was shut down at 2026-09-28 21:50:21 UTC
2026-09-28 21:50:22.825 UTC [13879] LOG: database system is ready to accept connections
I20260928 21:50:23.692234 3075 mini_postgres.cc:96] Postgres bound to 59809
2026-09-28 21:50:23.697 UTC [13890] FATAL: database "slave" does not exist
127.3.0.212:59809 - accepting connections
I20260928 21:50:23.698194 3075 mini_ranger.cc:161] Starting Ranger...
I20260928 21:50:23.711534 3075 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:50:24.099814 3075 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:50:24.099879 3075 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-admin
2026-09-28 21:50:24,365 [I] DB FLAVOR :POSTGRES
2026-09-28 21:50:24,366 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:50:24,366 [I] Checking connection..
2026-09-28 21:50:24,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:50:24,605 [I] Checking connection passed.
2026-09-28 21:50:24,606 [I] --------- Verifying version history table ---------
2026-09-28 21:50:24,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:50:24,856 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:50:24,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:50:25,079 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:50:25,080 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:50:25,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:50:25.300 UTC [13954] WARNING: there is no transaction in progress
2026-09-28 21:50:25,306 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:50:25,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:50:25,536 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:50:25,537 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:50:25,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:50:25,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:50:26,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jrck.c.gcp-upstream.internal','N') ;"
2026-09-28 21:50:26,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:50:26,467 [I] Table x_portal_user does not exist in database ranger
2026-09-28 21:50:26,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:50:26,706 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 21:50:26,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:50:26,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:50:27,184 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 21:50:27,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 21:50:28.037 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.053 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.069 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.082 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.096 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.115 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.175 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.181 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.194 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.203 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.212 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.225 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.233 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.239 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.244 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28.250 UTC [14062] WARNING: there is no transaction in progress
2026-09-28 21:50:28,773 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 21:50:28,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 21:50:28,997 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 21:50:28,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:50:29,219 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 21:50:29,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:50:29,459 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 21:50:29,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:50:29,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:50:29,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jrck.c.gcp-upstream.internal';"
2026-09-28 21:50:30,191 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 21:50:30,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 21:50:30,430 [I] DB_PATCHES have already been applied
I20260928 21:50:30.435388 3075 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksdiuc7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 21:50:30.435457 3075 mini_ranger.cc:193] Using host: 127.3.0.212
I20260928 21:50:30.438238 3075 mini_ranger.cc:227] Ranger admin URL: http://127.3.0.212:52521
Sep 28, 2026 9:50:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:50:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:50:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:50:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:50:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:50:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:50:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 21:50:31.426674 3075 mini_ranger.cc:160] Time spent starting Ranger: real 7.728s user 0.000s sys 0.008s
Sep 28, 2026 9:50:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:50:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 9:50:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:50:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:50:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 21:50:52.604475 3075 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.0.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.0.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/rangerkms_127.3.0.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/rangerkms_127.3.0.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.0.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.0.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.0.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/HTTP_127.3.0.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.0.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/HTTP_127.3.0.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 21:50:52 dist-test-slave-jrck krb5kdc[13849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632252, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 21:50:52.676726 3075 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 21:50:52.685307 3075 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 21:50:53.383947 3075 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 21:50:53.384030 3075 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-kms
I20260928 21:50:53.384044 3075 mini_ranger_kms.cc:227] Using postgres at 127.3.0.212:59809
2026-09-28 21:50:53,541 [I] DB FLAVOR :POSTGRES
2026-09-28 21:50:53,541 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:50:53,541 [I] Checking connection
2026-09-28 21:50:53,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 21:50:53,765 [I] connection success
2026-09-28 21:50:53,765 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 21:50:53,765 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 21:50:53,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 21:50:53,990 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 21:50:53,990 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:50:53,990 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 21:50:53,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:59809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 21:50:54,239 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 21:50:54.691964 3075 mini_ranger_kms.cc:310] Created kms service
I20260928 21:50:54.860117 3075 mini_ranger_kms.cc:326] Created kudu user
I20260928 21:50:54.902802 3075 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 21:50:54 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632252, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:50:55.281630 3075 mini_ranger_kms.cc:384] Added ranger policy
I20260928 21:50:55.281739 3075 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 21:50:55.281760 3075 mini_ranger_kms.cc:240] Using host: 127.3.0.212
I20260928 21:50:55.285367 3075 mini_ranger_kms.cc:276] Ranger KMS PID: 14242
I20260928 21:50:55.285470 3075 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.0.212:60273
Sep 28, 2026 9:50:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:50:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:50:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:50:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:50:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:50:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 21:50:57.318679 3075 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.642s user 0.004s sys 0.136s
I20260928 21:50:57.318770 3075 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 21:50:57.318851 3075 mini_ranger_kms.cc:401] 127.3.0.212:60273/kms/v1/keys
Sep 28 21:50:57 dist-test-slave-jrck krb5kdc[13849](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632257, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.0.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 21:50:57 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632257, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.0.212@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
Sep 28 21:50:58 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632257, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.0.212@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
Sep 28 21:51:00 dist-test-slave-jrck krb5kdc[13849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:51:00.390754 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:40703
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:40703
--ranger_config_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:60273/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:00.471778 14293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:00.471997 14293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:00.472016 14293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:00.473225 14293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:51:00.473271 14293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:00.473285 14293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:00.473295 14293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:51:00.473306 14293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:51:00.474897 14293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.0.212:60273/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:40703
--ranger_config_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:40703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:00.475122 14293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:00.475342 14293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:51:00.480574 14293 server_base.cc:1034] running on GCE node
W20260928 21:51:00.480878 14301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:00.480954 14299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:00.480932 14298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:00.481349 14293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:00.481694 14293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:00.482889 14293 hybrid_clock.cc:648] HybridClock initialized: now 1790632260482871 us; error 34 us; skew 500 ppm
Sep 28 21:51:00 dist-test-slave-jrck krb5kdc[13849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:00.490996 14293 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:51:00.491520 14293 webserver.cc:460] Webserver started at http://127.3.0.254:36547/ using document root <none> and password file <none>
I20260928 21:51:00.491758 14293 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:00.491802 14293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:00.491925 14293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:51:00 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632260, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:00.538100 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "0901228c0a08463cab03e28475406ab9"
format_stamp: "Formatted at 2026-09-28 21:51:00 on dist-test-slave-jrck"
server_key: "69578e0c20101b9c21bda14c98c909da"
server_key_iv: "8c6e667bdacb6b49db285475d4ff06cb"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:00.538455 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "0901228c0a08463cab03e28475406ab9"
format_stamp: "Formatted at 2026-09-28 21:51:00 on dist-test-slave-jrck"
server_key: "69578e0c20101b9c21bda14c98c909da"
server_key_iv: "8c6e667bdacb6b49db285475d4ff06cb"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:00.540246 14293 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:51:00.552623 14308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:00.552835 14293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:00.552954 14293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "0901228c0a08463cab03e28475406ab9"
format_stamp: "Formatted at 2026-09-28 21:51:00 on dist-test-slave-jrck"
server_key: "69578e0c20101b9c21bda14c98c909da"
server_key_iv: "8c6e667bdacb6b49db285475d4ff06cb"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:00.553054 14293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:00.568496 14293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:00.570389 14293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:00.570535 14293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:00.575883 14293 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:40703
I20260928 21:51:00.575918 14360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:40703 every 8 connection(s)
I20260928 21:51:00.576331 14293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:51:00.577025 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:00.579301 14361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9: Bootstrap starting.
I20260928 21:51:00.579859 14361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:00.580092 14361 log.cc:824] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:00.580754 14361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9: No bootstrap required, opened a new log
I20260928 21:51:00.582707 14361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901228c0a08463cab03e28475406ab9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40703 } }
I20260928 21:51:00.582849 14361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:00.582877 14361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0901228c0a08463cab03e28475406ab9, State: Initialized, Role: FOLLOWER
I20260928 21:51:00.583006 14361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901228c0a08463cab03e28475406ab9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40703 } }
I20260928 21:51:00.583081 14361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:51:00.583112 14361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:51:00.583158 14361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:00.583660 14361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901228c0a08463cab03e28475406ab9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40703 } }
I20260928 21:51:00.583753 14361 leader_election.cc:302] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0901228c0a08463cab03e28475406ab9; no voters:
I20260928 21:51:00.583902 14361 leader_election.cc:288] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:51:00.583942 14364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:51:00.584070 14364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 0901228c0a08463cab03e28475406ab9, State: Running, Role: LEADER
I20260928 21:51:00.584223 14361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:51:00.584225 14364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901228c0a08463cab03e28475406ab9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40703 } }
I20260928 21:51:00.584679 14366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0901228c0a08463cab03e28475406ab9. Latest consensus state: current_term: 1 leader_uuid: "0901228c0a08463cab03e28475406ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901228c0a08463cab03e28475406ab9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40703 } } }
I20260928 21:51:00.584689 14365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0901228c0a08463cab03e28475406ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901228c0a08463cab03e28475406ab9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40703 } } }
I20260928 21:51:00.584837 14366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:00.584856 14365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:00.585067 14373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:51:00.585512 14373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:51:00.585780 14361 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:51:00.586503 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 14293
I20260928 21:51:00.586627 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:51:00.586808 3075 external_mini_cluster.cc:1428] Setting key 437da4260a3a31b60b978b66b2e323f0
Sep 28 21:51:00 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:00.600052 14381 mvcc.cc:205] Tried to move back new op lower bound from 7334429739374223360 to 7334429739353591808. Current Snapshot: MvccSnapshot[applied={T|T < 7334429739374223360}]
I20260928 21:51:00.600749 14373 catalog_manager.cc:1347] Generated new cluster ID: 631060f8872f4c1ea166628723e5a787
I20260928 21:51:00.600817 14373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:51:00.605950 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:00.588053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.3.0.254:40703)
0928 21:51:00.592671 (+ 4618us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:00.592681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:00.592743 (+ 62us) server_negotiation.cc:407] Connection header received
0928 21:51:00.592997 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:00.593010 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:00.593083 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:00.594067 (+ 984us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:00.594187 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:00.595625 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:00.596755 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:00.597378 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:00.599817 (+ 2439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:00.600365 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:00.600370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:00.600370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:00.600383 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:00.603855 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:00.604266 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:00.604269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:00.604270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:00.604367 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:00.604602 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:00.604603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:00.604603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:00.604920 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:00.605505 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:00.605589 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 21:51:00.605718 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4514,"spinlock_wait_cycles":896,"thread_start_us":845,"threads_started":1}
I20260928 21:51:00.624575 14373 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:51:00.625470 14373 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:51:00.646704 14373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9: Generated new TSK 0
I20260928 21:51:00.647009 14373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:51:01.327190 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:60273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40703
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:01.433481 14406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:01.433691 14406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:01.433723 14406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:01.434948 14406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:01.435000 14406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:01.435038 14406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:51:01.436733 14406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.0.212:60273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:01.436982 14406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:01.437209 14406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:01.443303 14414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:01.443442 14406 server_base.cc:1034] running on GCE node
W20260928 21:51:01.443405 14411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:01.443303 14412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:01.443846 14406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:01.444098 14406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:01.445259 14406 hybrid_clock.cc:648] HybridClock initialized: now 1790632261445231 us; error 46 us; skew 500 ppm
Sep 28 21:51:01 dist-test-slave-jrck krb5kdc[13849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:01.451303 14406 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:51:01.451849 14406 webserver.cc:460] Webserver started at http://127.3.0.193:42325/ using document root <none> and password file <none>
I20260928 21:51:01.452072 14406 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:01.452145 14406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:01.452283 14406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:51:01 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:01.466262 14406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "5f06b3aa3c39422f903849f8eac10cb8"
format_stamp: "Formatted at 2026-09-28 21:51:01 on dist-test-slave-jrck"
server_key: "697255d33370898f215c2a9df255044d"
server_key_iv: "ffa220b9d85f984d082c4052fd7ed14e"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:01.466717 14406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "5f06b3aa3c39422f903849f8eac10cb8"
format_stamp: "Formatted at 2026-09-28 21:51:01 on dist-test-slave-jrck"
server_key: "697255d33370898f215c2a9df255044d"
server_key_iv: "ffa220b9d85f984d082c4052fd7ed14e"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:01.468838 14406 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:51:01.485666 14423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:01.486001 14406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260928 21:51:01.486146 14406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "5f06b3aa3c39422f903849f8eac10cb8"
format_stamp: "Formatted at 2026-09-28 21:51:01 on dist-test-slave-jrck"
server_key: "697255d33370898f215c2a9df255044d"
server_key_iv: "ffa220b9d85f984d082c4052fd7ed14e"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:01.486274 14406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:01.510027 14406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:01.511947 14406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:01.512092 14406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:01.512293 14406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:01.512677 14406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:01.512715 14406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:01.512744 14406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:01.512782 14406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:01.519622 14406 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:45519
I20260928 21:51:01.520008 14406 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:51:01.520280 14536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:45519 every 8 connection(s)
I20260928 21:51:01.524755 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 14406
I20260928 21:51:01.524865 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:51:01.525095 3075 external_mini_cluster.cc:1428] Setting key 43587ff9195aa3a50b7600b7d87f2e67
Sep 28 21:51:01 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:01.530336 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:01.521022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:60583 (local address 127.3.0.254:40703)
0928 21:51:01.521173 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:01.521175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:01.521274 (+ 99us) server_negotiation.cc:407] Connection header received
0928 21:51:01.521766 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:01.521768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:01.521782 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:01.521884 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:01.522423 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:01.523236 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:01.524387 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:01.524758 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:01.524840 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:01.527315 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:01.527316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:01.527317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:01.527323 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:01.528890 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:01.529321 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:01.529322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:01.529322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:01.529375 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:01.529631 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:01.529632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:01.529632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:01.529779 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:01.529842 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:01.530154 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 21:51:01.530238 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260928 21:51:01.530490 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:01.520956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:40703 (local address 127.3.0.193:60583)
0928 21:51:01.521131 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:01.521175 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:01.521654 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:01.521938 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:01.521941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:01.522123 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:01.522322 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:01.522327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:01.523354 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:01.523356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:01.524286 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:01.524289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:01.524905 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:01.524906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:01.525059 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:01.525628 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:01.525636 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:01.527216 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:01.528985 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:01.528993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:01.528996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:01.529251 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:01.529444 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:01.529445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:01.529445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:01.529558 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:01.529848 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:01.529855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:01.530009 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 21:51:01.530113 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:51:01.530262 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260928 21:51:01.530959 14537 heartbeater.cc:347] Connected to a master server at 127.3.0.254:40703
I20260928 21:51:01.531064 14537 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:01.531241 14537 heartbeater.cc:511] Master 127.3.0.254:40703 requested a full tablet report, sending...
I20260928 21:51:01.531754 14325 ts_manager.cc:194] Registered new tserver with Master: 5f06b3aa3c39422f903849f8eac10cb8 (127.3.0.193:45519)
I20260928 21:51:01.532699 14325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:60583
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:51:01.555145 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:60273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40703
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:51:01 dist-test-slave-jrck krb5kdc[13849](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 21:51:01.678227 14543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:01.678593 14543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:01.678694 14543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:01.680475 14543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:01.680588 14543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:01.680684 14543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:51:01.682919 14543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.0.212:60273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:01.683368 14543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:01.683696 14543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:01.690073 14551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:01.690375 14550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:01.690402 14553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:01.690636 14543 server_base.cc:1034] running on GCE node
I20260928 21:51:01.690819 14543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:01.691056 14543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:01.692220 14543 hybrid_clock.cc:648] HybridClock initialized: now 1790632261692199 us; error 33 us; skew 500 ppm
Sep 28 21:51:01 dist-test-slave-jrck krb5kdc[13849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:01.695514 14543 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:51:01.696084 14543 webserver.cc:460] Webserver started at http://127.3.0.194:42505/ using document root <none> and password file <none>
I20260928 21:51:01.696394 14543 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:01.696523 14543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:01.696702 14543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:51:01 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:01.717633 14543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "cae54ac5e53c46d6a014dec187fb8145"
format_stamp: "Formatted at 2026-09-28 21:51:01 on dist-test-slave-jrck"
server_key: "5cb457efbc7b8173cbdf614bcdd63388"
server_key_iv: "6709eed7f314605e30acfdde89be65b2"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:01.718170 14543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "cae54ac5e53c46d6a014dec187fb8145"
format_stamp: "Formatted at 2026-09-28 21:51:01 on dist-test-slave-jrck"
server_key: "5cb457efbc7b8173cbdf614bcdd63388"
server_key_iv: "6709eed7f314605e30acfdde89be65b2"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:01.721314 14543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:51:01.734402 14560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:01.734851 14543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:01.735054 14543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "cae54ac5e53c46d6a014dec187fb8145"
format_stamp: "Formatted at 2026-09-28 21:51:01 on dist-test-slave-jrck"
server_key: "5cb457efbc7b8173cbdf614bcdd63388"
server_key_iv: "6709eed7f314605e30acfdde89be65b2"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:01.735180 14543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:01.754106 14543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:01.756549 14543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:01.756765 14543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:01.757097 14543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:01.757545 14543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:01.757640 14543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:01.757720 14543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:01.757778 14543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:01.765514 14543 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:41077
I20260928 21:51:01.766048 14543 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:51:01.769373 14674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:41077 every 8 connection(s)
I20260928 21:51:01.775872 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 14543
I20260928 21:51:01.775974 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:51:01.776141 3075 external_mini_cluster.cc:1428] Setting key 769e7dc59651ab59e1f54b61e7fc19a2
Sep 28 21:51:01 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:01.784930 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:01.773633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:39733 (local address 127.3.0.254:40703)
0928 21:51:01.773677 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:01.773679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:01.773908 (+ 229us) server_negotiation.cc:407] Connection header received
0928 21:51:01.774713 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:01.774714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:01.774727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:01.774757 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:01.775613 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:01.776332 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:01.777741 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:01.778011 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:01.778059 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:01.781545 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:01.781547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:01.781547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:01.781556 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:01.783274 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:01.783750 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:01.783751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:01.783752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:01.783809 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:01.784071 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:01.784072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:01.784072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:01.784238 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:01.784280 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:01.784741 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 21:51:01.784830 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:51:01.785497 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:01.773342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:40703 (local address 127.3.0.194:39733)
0928 21:51:01.773763 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:01.773813 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:01.774588 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:01.774943 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:01.774947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:01.775209 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:01.775494 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:01.775502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:01.776444 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:01.776445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:01.777646 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:01.777650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:01.778143 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:01.778144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:01.778282 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:01.778869 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:01.778880 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:01.781426 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:01.783365 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:01.783371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:01.783374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:01.783663 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:01.783871 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:01.783871 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:01.783872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:01.784000 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:01.784343 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:01.784350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:01.784588 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 21:51:01.785026 (+ 438us) client_negotiation.cc:240] Negotiation successful
0928 21:51:01.785207 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":89,"threads_started":1}
I20260928 21:51:01.786159 14675 heartbeater.cc:347] Connected to a master server at 127.3.0.254:40703
I20260928 21:51:01.786266 14675 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:01.786455 14675 heartbeater.cc:511] Master 127.3.0.254:40703 requested a full tablet report, sending...
I20260928 21:51:01.786827 14325 ts_manager.cc:194] Registered new tserver with Master: cae54ac5e53c46d6a014dec187fb8145 (127.3.0.194:41077)
I20260928 21:51:01.787299 14325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:39733
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:51:01.807303 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:60273/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40703
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:01.931038 14680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:01.931432 14680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:01.931536 14680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:01.933384 14680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:01.933506 14680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:01.933606 14680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:51:01.935889 14680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.0.212:60273/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:01.936371 14680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:01.936724 14680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:01.947377 14685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:01.947790 14688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:01.948722 14680 server_base.cc:1034] running on GCE node
W20260928 21:51:01.948958 14686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:01.949219 14680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:01.949478 14680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:01.950660 14680 hybrid_clock.cc:648] HybridClock initialized: now 1790632261950639 us; error 35 us; skew 500 ppm
Sep 28 21:51:01 dist-test-slave-jrck krb5kdc[13849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:01.953819 14680 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:51:01.954303 14680 webserver.cc:460] Webserver started at http://127.3.0.195:44751/ using document root <none> and password file <none>
I20260928 21:51:01.954558 14680 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:01.954608 14680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:01.954735 14680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:51:01 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:01.973455 14680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "81e48845d8a14ae7887b316175ede8b3"
format_stamp: "Formatted at 2026-09-28 21:51:01 on dist-test-slave-jrck"
server_key: "aa74506b1f85970c3a2e5d67d35b1c02"
server_key_iv: "7c80703d981212eca540c6d27840d897"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:01.973835 14680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "81e48845d8a14ae7887b316175ede8b3"
format_stamp: "Formatted at 2026-09-28 21:51:01 on dist-test-slave-jrck"
server_key: "aa74506b1f85970c3a2e5d67d35b1c02"
server_key_iv: "7c80703d981212eca540c6d27840d897"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:01.976135 14680 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:51:01.987401 14695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:01.987629 14680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:51:01.987742 14680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "81e48845d8a14ae7887b316175ede8b3"
format_stamp: "Formatted at 2026-09-28 21:51:01 on dist-test-slave-jrck"
server_key: "aa74506b1f85970c3a2e5d67d35b1c02"
server_key_iv: "7c80703d981212eca540c6d27840d897"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:01.987843 14680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:02.000160 14680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:02.002378 14680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:02.002523 14680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:02.002771 14680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:02.003151 14680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:02.003191 14680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:02.003221 14680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:02.003260 14680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:02.009943 14680 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:39311
I20260928 21:51:02.017263 14680 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:51:02.010130 14808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:39311 every 8 connection(s)
Sep 28 21:51:02 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:02.026777 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 14680
I20260928 21:51:02.026885 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:51:02.027113 3075 external_mini_cluster.cc:1428] Setting key 805e7a4135afbd261004774df9713628
I20260928 21:51:02.030577 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.018862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:38927 (local address 127.3.0.254:40703)
0928 21:51:02.018909 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:02.018911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:02.019140 (+ 229us) server_negotiation.cc:407] Connection header received
0928 21:51:02.019908 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:02.019909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:02.019922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:02.019953 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:02.020914 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.021687 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.023140 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.023432 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.023472 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.026042 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:02.026043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:02.026044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:02.026052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:02.027837 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:02.028922 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:02.028923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:02.028924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:02.028988 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:02.029783 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:02.029784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:02.029785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:02.029930 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:02.029984 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:02.030373 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 21:51:02.030464 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:51:02.031342 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.018569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:40703 (local address 127.3.0.195:38927)
0928 21:51:02.019003 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:02.019054 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:02.019782 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:02.020157 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:02.020162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:02.020408 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:02.020801 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.020809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.021799 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.021800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.023038 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.023041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.023569 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.023570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.023705 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.024243 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:02.024254 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:02.025932 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:02.028133 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:02.028140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:02.028142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:02.028830 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:02.029062 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:02.029063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:02.029063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:02.029694 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:02.030018 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:02.030024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:02.030211 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 21:51:02.030929 (+ 718us) client_negotiation.cc:240] Negotiation successful
0928 21:51:02.031101 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":89,"threads_started":1}
I20260928 21:51:02.033018 14809 heartbeater.cc:347] Connected to a master server at 127.3.0.254:40703
I20260928 21:51:02.033118 14809 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:02.033314 14809 heartbeater.cc:511] Master 127.3.0.254:40703 requested a full tablet report, sending...
I20260928 21:51:02.033746 14325 ts_manager.cc:194] Registered new tserver with Master: 81e48845d8a14ae7887b316175ede8b3 (127.3.0.195:39311)
I20260928 21:51:02.034236 14325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:38927
I20260928 21:51:02.039425 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:51:02.060565 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.048779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.3.0.254:40703)
0928 21:51:02.048876 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:02.048879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:02.049130 (+ 251us) server_negotiation.cc:407] Connection header received
0928 21:51:02.049242 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:02.049244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:02.049254 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:02.049283 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:02.050099 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.050867 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.052363 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.052764 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.052837 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.055001 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:02.055003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:02.055004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:02.055011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:02.058876 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:02.059380 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:02.059381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:02.059382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:02.059446 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:02.059913 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:02.059915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:02.059915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:02.060060 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:02.060138 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:02.060365 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 21:51:02.060444 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:51:02.063279 14325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:51:02.065644 14325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:51:02.087347 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.071031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:41077 (local address 127.0.0.1:46760)
0928 21:51:02.074175 (+ 3144us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:02.074184 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:02.074233 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:02.074593 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:02.074596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:02.074598 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:02.074667 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.074669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.076669 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.076670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.084597 (+ 7927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.084600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.086927 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.086928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.087095 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.087121 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:51:02.087198 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:51:02.087259 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3130,"thread_start_us":58,"threads_started":1}
I20260928 21:51:02.088430 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.071740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:39311 (local address 127.0.0.1:45704)
0928 21:51:02.077852 (+ 6112us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:02.077864 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:02.078132 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:02.078560 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:02.078561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:02.078563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:02.078721 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.078724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.080744 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.080745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.083709 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.083712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.088196 (+ 4484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.088197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.088262 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.088266 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:51:02.088312 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:51:02.088353 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6068,"thread_start_us":64,"threads_started":1}
I20260928 21:51:02.089159 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.070813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:45519 (local address 127.0.0.1:38008)
0928 21:51:02.073628 (+ 2815us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:02.073672 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:02.073736 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:02.074739 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:02.074740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:02.074742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:02.074779 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.074780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.081703 (+ 6923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.081704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.082855 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.082858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.088932 (+ 6074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.088933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.088997 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.089001 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:51:02.089049 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 21:51:02.089089 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2783,"thread_start_us":82,"threads_started":1}
I20260928 21:51:02.090009 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.071306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.3.0.193:45519)
0928 21:51:02.073860 (+ 2554us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:02.073866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:02.073875 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:51:02.073909 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:02.073912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:02.073927 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:02.073993 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:02.074851 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.081609 (+ 6758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.085727 (+ 4118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.088775 (+ 3048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.088843 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.088897 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:02.089892 (+ 995us) server_negotiation.cc:299] Negotiation successful
0928 21:51:02.089939 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2521,"thread_start_us":52,"threads_started":1}
I20260928 21:51:02.090370 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.071439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.3.0.194:41077)
0928 21:51:02.074121 (+ 2682us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:02.074126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:02.074374 (+ 248us) server_negotiation.cc:407] Connection header received
0928 21:51:02.074414 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:02.074416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:02.074431 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:02.074495 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:02.075444 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.076579 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.084714 (+ 8135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.085531 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.085601 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.085689 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:02.090236 (+ 4547us) server_negotiation.cc:299] Negotiation successful
0928 21:51:02.090288 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2647,"thread_start_us":88,"threads_started":1}
I20260928 21:51:02.090728 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.071932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.3.0.195:39311)
0928 21:51:02.074352 (+ 2420us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:02.074356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:02.078100 (+ 3744us) server_negotiation.cc:407] Connection header received
0928 21:51:02.078300 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:02.078303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:02.078321 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:02.078441 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:02.078822 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.079880 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.086387 (+ 6507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.088009 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.088083 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.088160 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:02.090560 (+ 2400us) server_negotiation.cc:299] Negotiation successful
0928 21:51:02.090648 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2382,"thread_start_us":74,"threads_started":1}
I20260928 21:51:02.091548 14471 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdae7e0223e4bce94098c799c0efba9 (DEFAULT_TABLE table=test-table [id=87274e1977a2443d94283a148531122e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:02.091936 14471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdae7e0223e4bce94098c799c0efba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:02.096247 14828 tablet_bootstrap.cc:492] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8: Bootstrap starting.
I20260928 21:51:02.096359 14602 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdae7e0223e4bce94098c799c0efba9 (DEFAULT_TABLE table=test-table [id=87274e1977a2443d94283a148531122e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:02.096736 14602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdae7e0223e4bce94098c799c0efba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:02.097045 14828 tablet_bootstrap.cc:654] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:02.097337 14828 log.cc:824] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:02.098070 14828 tablet_bootstrap.cc:492] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8: No bootstrap required, opened a new log
I20260928 21:51:02.098129 14828 ts_tablet_manager.cc:1397] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:51:02.099828 14725 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdae7e0223e4bce94098c799c0efba9 (DEFAULT_TABLE table=test-table [id=87274e1977a2443d94283a148531122e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:02.100109 14725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdae7e0223e4bce94098c799c0efba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:02.100560 14828 raft_consensus.cc:348] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e48845d8a14ae7887b316175ede8b3" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 39311 } } peers { permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 } } peers { permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 } }
I20260928 21:51:02.100720 14828 raft_consensus.cc:374] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:02.100754 14828 raft_consensus.cc:729] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f06b3aa3c39422f903849f8eac10cb8, State: Initialized, Role: FOLLOWER
I20260928 21:51:02.100886 14828 consensus_queue.cc:260] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e48845d8a14ae7887b316175ede8b3" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 39311 } } peers { permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 } } peers { permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 } }
I20260928 21:51:02.101174 14828 ts_tablet_manager.cc:1428] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:51:02.101277 14537 heartbeater.cc:503] Master 127.3.0.254:40703 was elected leader, sending a full tablet report...
I20260928 21:51:02.103070 14831 tablet_bootstrap.cc:492] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: Bootstrap starting.
I20260928 21:51:02.104054 14831 tablet_bootstrap.cc:654] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:02.104959 14831 log.cc:824] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:02.105875 14831 tablet_bootstrap.cc:492] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: No bootstrap required, opened a new log
I20260928 21:51:02.105939 14831 ts_tablet_manager.cc:1397] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:51:02.107821 14831 raft_consensus.cc:348] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e48845d8a14ae7887b316175ede8b3" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 39311 } } peers { permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 } } peers { permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 } }
I20260928 21:51:02.107977 14831 raft_consensus.cc:374] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:02.108007 14831 raft_consensus.cc:729] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e48845d8a14ae7887b316175ede8b3, State: Initialized, Role: FOLLOWER
I20260928 21:51:02.108119 14831 consensus_queue.cc:260] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e48845d8a14ae7887b316175ede8b3" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 39311 } } peers { permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 } } peers { permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 } }
I20260928 21:51:02.108363 14831 ts_tablet_manager.cc:1428] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:51:02.108598 14809 heartbeater.cc:503] Master 127.3.0.254:40703 was elected leader, sending a full tablet report...
I20260928 21:51:02.108619 14832 tablet_bootstrap.cc:492] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: Bootstrap starting.
I20260928 21:51:02.109637 14832 tablet_bootstrap.cc:654] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:02.112870 14832 log.cc:824] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:02.115298 14832 tablet_bootstrap.cc:492] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: No bootstrap required, opened a new log
I20260928 21:51:02.115374 14832 ts_tablet_manager.cc:1397] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 21:51:02.117480 14832 raft_consensus.cc:348] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e48845d8a14ae7887b316175ede8b3" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 39311 } } peers { permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 } } peers { permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 } }
I20260928 21:51:02.117671 14832 raft_consensus.cc:374] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:02.117707 14832 raft_consensus.cc:729] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae54ac5e53c46d6a014dec187fb8145, State: Initialized, Role: FOLLOWER
I20260928 21:51:02.117833 14832 consensus_queue.cc:260] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e48845d8a14ae7887b316175ede8b3" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 39311 } } peers { permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 } } peers { permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 } }
I20260928 21:51:02.118113 14832 ts_tablet_manager.cc:1428] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:51:02.118256 14675 heartbeater.cc:503] Master 127.3.0.254:40703 was elected leader, sending a full tablet report...
Sep 28 21:51:02 dist-test-slave-jrck krb5kdc[13849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632261, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:02.240130 14834 raft_consensus.cc:482] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:51:02.240274 14834 raft_consensus.cc:504] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e48845d8a14ae7887b316175ede8b3" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 39311 } } peers { permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 } } peers { permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 } }
I20260928 21:51:02.240738 14834 leader_election.cc:288] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f06b3aa3c39422f903849f8eac10cb8 (127.3.0.193:45519), cae54ac5e53c46d6a014dec187fb8145 (127.3.0.194:41077)
I20260928 21:51:02.245237 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.241091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:45519 (local address 127.3.0.195:53441)
0928 21:51:02.241506 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:02.241519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:02.241565 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:02.241921 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:02.241922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:02.241924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:02.241994 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.241996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.242923 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.242924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.244130 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.244133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.244940 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.244941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.245069 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.245073 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:51:02.245120 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:51:02.245162 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394}
I20260928 21:51:02.245937 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.241671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:53441 (local address 127.3.0.193:45519)
0928 21:51:02.241710 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:02.241712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:02.241719 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:51:02.241745 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:02.241746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:02.241756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:02.241797 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:02.242087 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.242822 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.244233 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.244830 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.244897 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.244910 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:02.245823 (+ 913us) server_negotiation.cc:299] Negotiation successful
0928 21:51:02.245873 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:51:02.246230 14491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdae7e0223e4bce94098c799c0efba9" candidate_uuid: "81e48845d8a14ae7887b316175ede8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f06b3aa3c39422f903849f8eac10cb8" is_pre_election: true
I20260928 21:51:02.246393 14491 raft_consensus.cc:2454] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e48845d8a14ae7887b316175ede8b3 in term 0.
I20260928 21:51:02.247337 14698 leader_election.cc:302] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f06b3aa3c39422f903849f8eac10cb8, 81e48845d8a14ae7887b316175ede8b3; no voters:
I20260928 21:51:02.247529 14834 raft_consensus.cc:2781] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:51:02.247586 14834 raft_consensus.cc:482] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:51:02.247608 14834 raft_consensus.cc:3037] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:02.248525 14834 raft_consensus.cc:504] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e48845d8a14ae7887b316175ede8b3" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 39311 } } peers { permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 } } peers { permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 } }
I20260928 21:51:02.248647 14834 leader_election.cc:288] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [CANDIDATE]: Term 1 election: Requested vote from peers 5f06b3aa3c39422f903849f8eac10cb8 (127.3.0.193:45519), cae54ac5e53c46d6a014dec187fb8145 (127.3.0.194:41077)
I20260928 21:51:02.248919 14491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdae7e0223e4bce94098c799c0efba9" candidate_uuid: "81e48845d8a14ae7887b316175ede8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f06b3aa3c39422f903849f8eac10cb8"
I20260928 21:51:02.248997 14491 raft_consensus.cc:3037] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:02.249809 14491 raft_consensus.cc:2454] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e48845d8a14ae7887b316175ede8b3 in term 1.
I20260928 21:51:02.249929 14698 leader_election.cc:302] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f06b3aa3c39422f903849f8eac10cb8, 81e48845d8a14ae7887b316175ede8b3; no voters:
I20260928 21:51:02.250705 14834 raft_consensus.cc:2781] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:51:02.250856 14834 raft_consensus.cc:686] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 1 LEADER]: Becoming Leader. State: Replica: 81e48845d8a14ae7887b316175ede8b3, State: Running, Role: LEADER
I20260928 21:51:02.250949 14834 consensus_queue.cc:237] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e48845d8a14ae7887b316175ede8b3" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 39311 } } peers { permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 } } peers { permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 } }
I20260928 21:51:02.251807 14325 catalog_manager.cc:5495] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81e48845d8a14ae7887b316175ede8b3 (127.3.0.195). New cstate: current_term: 1 leader_uuid: "81e48845d8a14ae7887b316175ede8b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e48845d8a14ae7887b316175ede8b3" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 39311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 } health_report { overall_health: UNKNOWN } } }
I20260928 21:51:02.252682 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.241194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:41077 (local address 127.3.0.195:46567)
0928 21:51:02.245303 (+ 4109us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:02.245313 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:02.245348 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:02.246273 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:02.246274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:02.246276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:02.246308 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.246309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.248875 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.248876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.249835 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:02.249838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.252428 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.252430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:02.252512 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.252516 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:51:02.252575 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:51:02.252613 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4099,"thread_start_us":100,"threads_started":1}
I20260928 21:51:02.253319 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.241854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:46567 (local address 127.3.0.194:41077)
0928 21:51:02.241890 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:02.241891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:02.245397 (+ 3506us) server_negotiation.cc:407] Connection header received
0928 21:51:02.245426 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:02.245427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:02.245438 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:02.245479 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:02.246387 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.247088 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.250061 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.250596 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.250664 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.250676 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:02.253206 (+ 2530us) server_negotiation.cc:299] Negotiation successful
0928 21:51:02.253254 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:51:02.253602 14629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdae7e0223e4bce94098c799c0efba9" candidate_uuid: "81e48845d8a14ae7887b316175ede8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae54ac5e53c46d6a014dec187fb8145" is_pre_election: true
I20260928 21:51:02.253741 14629 raft_consensus.cc:2454] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81e48845d8a14ae7887b316175ede8b3 in term 0.
I20260928 21:51:02.253858 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdae7e0223e4bce94098c799c0efba9" candidate_uuid: "81e48845d8a14ae7887b316175ede8b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae54ac5e53c46d6a014dec187fb8145"
I20260928 21:51:02.253918 14628 raft_consensus.cc:3037] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:02.254741 14628 raft_consensus.cc:2454] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81e48845d8a14ae7887b316175ede8b3 in term 1.
W20260928 21:51:02.268304 14810 tablet.cc:2369] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:51:02.269428 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.264934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.3.0.195:39311)
0928 21:51:02.265147 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:02.265150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:02.265157 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:51:02.265190 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:02.265190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:02.265212 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:02.265272 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:51:02.265471 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.266210 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.267331 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:02.268123 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:02.268186 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:02.268848 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:51:02.268978 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:51:02.269073 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:02.269324 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 21:51:02.269345 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
W20260928 21:51:02.270823 14538 tablet.cc:2369] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:51:02.271656 14628 raft_consensus.cc:1261] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145 [term 1 FOLLOWER]: Refusing update from remote peer 81e48845d8a14ae7887b316175ede8b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:51:02.272068 14838 consensus_queue.cc:1035] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae54ac5e53c46d6a014dec187fb8145" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:51:02.273010 14676 tablet.cc:2369] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:51:02.274310 14491 raft_consensus.cc:1261] T 9fdae7e0223e4bce94098c799c0efba9 P 5f06b3aa3c39422f903849f8eac10cb8 [term 1 FOLLOWER]: Refusing update from remote peer 81e48845d8a14ae7887b316175ede8b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:51:02.274621 14838 consensus_queue.cc:1035] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f06b3aa3c39422f903849f8eac10cb8" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:51:02.278537 14841 mvcc.cc:205] Tried to move back new op lower bound from 7334429746261753856 to 7334429746180526080. Current Snapshot: MvccSnapshot[applied={T|T < 7334429746261753856}]
I20260928 21:51:02.280864 14844 mvcc.cc:205] Tried to move back new op lower bound from 7334429746261753856 to 7334429746180526080. Current Snapshot: MvccSnapshot[applied={T|T < 7334429746261753856}]
I20260928 21:51:02.286460 14325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:51:02.286566 14325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:51:02.287345 14325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0901228c0a08463cab03e28475406ab9: Sending DeleteTablet for 3 replicas of tablet 9fdae7e0223e4bce94098c799c0efba9
I20260928 21:51:02.287700 14471 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdae7e0223e4bce94098c799c0efba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:02 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:38008
I20260928 21:51:02.287700 14602 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdae7e0223e4bce94098c799c0efba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:02 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:46760
I20260928 21:51:02.288169 14725 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdae7e0223e4bce94098c799c0efba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:02 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:45704
I20260928 21:51:02.288678 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 14406
I20260928 21:51:02.296106 14851 tablet_replica.cc:330] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: stopping tablet replica
I20260928 21:51:02.296361 14851 raft_consensus.cc:2229] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:51:02.296669 14851 raft_consensus.cc:2258] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:51:02.297260 14851 ts_tablet_manager.cc:1905] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:51:02.300558 14853 tablet_replica.cc:330] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: stopping tablet replica
I20260928 21:51:02.300681 14853 raft_consensus.cc:2229] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:51:02.300829 14853 raft_consensus.cc:2258] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:51:02.301332 14853 ts_tablet_manager.cc:1905] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:51:02.302906 14853 ts_tablet_manager.cc:1918] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:51:02.302994 14853 log.cc:1196] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/9fdae7e0223e4bce94098c799c0efba9
I20260928 21:51:02.303251 14853 ts_tablet_manager.cc:1939] T 9fdae7e0223e4bce94098c799c0efba9 P 81e48845d8a14ae7887b316175ede8b3: Deleting consensus metadata
I20260928 21:51:02.303678 14312 catalog_manager.cc:4841] TS 81e48845d8a14ae7887b316175ede8b3 (127.3.0.195:39311): tablet 9fdae7e0223e4bce94098c799c0efba9 (table test-table [id=87274e1977a2443d94283a148531122e]) successfully deleted
I20260928 21:51:02.304565 14851 ts_tablet_manager.cc:1918] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:51:02.304641 14851 log.cc:1196] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/9fdae7e0223e4bce94098c799c0efba9
I20260928 21:51:02.304860 14851 ts_tablet_manager.cc:1939] T 9fdae7e0223e4bce94098c799c0efba9 P cae54ac5e53c46d6a014dec187fb8145: Deleting consensus metadata
I20260928 21:51:02.305272 14311 catalog_manager.cc:4841] TS cae54ac5e53c46d6a014dec187fb8145 (127.3.0.194:41077): tablet 9fdae7e0223e4bce94098c799c0efba9 (table test-table [id=87274e1977a2443d94283a148531122e]) successfully deleted
W20260928 21:51:02.308722 14311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:45519 (error 108)
I20260928 21:51:02.316799 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 14543
I20260928 21:51:02.318809 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:02.318663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:45519 (local address 127.0.0.1:38022)
0928 21:51:02.318726 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:02.318768 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:45519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 21:51:02.319104 14311 catalog_manager.cc:4584] TS 5f06b3aa3c39422f903849f8eac10cb8 (127.3.0.193:45519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fdae7e0223e4bce94098c799c0efba9: Network error: Client connection negotiation failed: client connection to 127.3.0.193:45519: connect: Connection refused (error 111)
I20260928 21:51:02.322695 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 14680
I20260928 21:51:02.332854 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 14293
I20260928 21:51:02.339020 3075 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 21:51:02.673050 3075 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 21:51:02.673159 3075 mini_ranger.cc:66] Stopping Ranger...
I20260928 21:51:03.007505 3075 mini_ranger.cc:68] Stopped Ranger
2026-09-28 21:51:03.007 UTC [13879] LOG: received smart shutdown request
2026-09-28 21:51:03.010 UTC [13879] LOG: background worker "logical replication launcher" (PID 13888) exited with exit code 1
2026-09-28 21:51:03.011 UTC [13883] LOG: shutting down
2026-09-28 21:51:03.154 UTC [13879] LOG: database system is shut down
2026-09-28T21:51:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43721 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:51:03 dist-test-slave-jrck krb5kdc[14863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:51:03 dist-test-slave-jrck krb5kdc[14863](info): set up 2 sockets
Sep 28 21:51:03 dist-test-slave-jrck krb5kdc[14863](info): commencing operation
krb5kdc: starting...
W20260928 21:51:04.188207 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:51:04 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:51:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.0.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.0.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/rangeradmin_127.3.0.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/rangeradmin_127.3.0.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.0.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.0.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/rangerlookup_127.3.0.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/rangerlookup_127.3.0.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.0.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.0.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/HTTP_127.3.0.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/HTTP_127.3.0.212@KRBTEST.COM.keytab.
I20260928 21:51:04.336537 3075 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksdiuc7/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/postgres -l logfile start
2026-09-28 21:51:06.555 UTC [14893] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:51:06.555 UTC [14893] LOG: listening on IPv4 address "127.3.0.212", port 58847
2026-09-28 21:51:06.558 UTC [14893] LOG: listening on Unix socket "/tmp/.s.PGSQL.58847"
2026-09-28 21:51:06.582 UTC [14896] LOG: database system was shut down at 2026-09-28 21:51:05 UTC
2026-09-28 21:51:06.586 UTC [14893] LOG: database system is ready to accept connections
I20260928 21:51:07.448035 3075 mini_postgres.cc:96] Postgres bound to 58847
2026-09-28 21:51:07.453 UTC [14904] FATAL: database "slave" does not exist
127.3.0.212:58847 - accepting connections
I20260928 21:51:07.454099 3075 mini_ranger.cc:161] Starting Ranger...
I20260928 21:51:07.467813 3075 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:51:07.860195 3075 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:51:07.860275 3075 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-admin
2026-09-28 21:51:08,118 [I] DB FLAVOR :POSTGRES
2026-09-28 21:51:08,118 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:51:08,118 [I] Checking connection..
2026-09-28 21:51:08,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:51:08,370 [I] Checking connection passed.
2026-09-28 21:51:08,371 [I] --------- Verifying version history table ---------
2026-09-28 21:51:08,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:51:08,626 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:51:08,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:51:08,880 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:51:08,880 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:51:08,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:51:09.125 UTC [14968] WARNING: there is no transaction in progress
2026-09-28 21:51:09,131 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:51:09,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:51:09,370 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:51:09,370 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:51:09,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:51:09,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:51:09,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jrck.c.gcp-upstream.internal','N') ;"
2026-09-28 21:51:10,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:51:10,317 [I] Table x_portal_user does not exist in database ranger
2026-09-28 21:51:10,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:51:10,569 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 21:51:10,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:51:10,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:51:11,067 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 21:51:11,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 21:51:11.949 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:11.964 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:11.978 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:11.992 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.009 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.023 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.083 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.089 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.102 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.111 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.121 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.133 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.141 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.147 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.152 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12.158 UTC [15076] WARNING: there is no transaction in progress
2026-09-28 21:51:12,676 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 21:51:12,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 21:51:12,909 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 21:51:12,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:51:13,144 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 21:51:13,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:51:13,396 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 21:51:13,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:51:13,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:51:13,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jrck.c.gcp-upstream.internal';"
2026-09-28 21:51:14,121 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 21:51:14,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 21:51:14,355 [I] DB_PATCHES have already been applied
I20260928 21:51:14.360993 3075 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksdiuc7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 21:51:14.361073 3075 mini_ranger.cc:193] Using host: 127.3.0.212
I20260928 21:51:14.363708 3075 mini_ranger.cc:227] Ranger admin URL: http://127.3.0.212:45889
Sep 28, 2026 9:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:51:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:51:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 21:51:15.366560 3075 mini_ranger.cc:160] Time spent starting Ranger: real 7.912s user 0.000s sys 0.008s
Sep 28, 2026 9:51:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:51:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 9:51:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:51:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:51:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 21:51:36.876647 3075 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.0.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.0.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/rangerkms_127.3.0.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.0.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/rangerkms_127.3.0.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.0.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.0.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.0.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/HTTP_127.3.0.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.0.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/HTTP_127.3.0.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 21:51:36 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632296, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 21:51:36.956082 3075 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 21:51:36.965051 3075 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 21:51:37.659354 3075 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 21:51:37.659431 3075 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-kms
I20260928 21:51:37.659447 3075 mini_ranger_kms.cc:227] Using postgres at 127.3.0.212:58847
2026-09-28 21:51:37,805 [I] DB FLAVOR :POSTGRES
2026-09-28 21:51:37,805 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:51:37,805 [I] Checking connection
2026-09-28 21:51:37,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 21:51:38,026 [I] connection success
2026-09-28 21:51:38,027 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 21:51:38,027 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 21:51:38,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 21:51:38,271 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 21:51:38,271 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:51:38,271 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 21:51:38,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.0.212:58847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 21:51:38,512 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 21:51:38.956997 3075 mini_ranger_kms.cc:310] Created kms service
I20260928 21:51:39.156739 3075 mini_ranger_kms.cc:326] Created kudu user
I20260928 21:51:39.199060 3075 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 21:51:39 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632296, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:39.582428 3075 mini_ranger_kms.cc:384] Added ranger policy
I20260928 21:51:39.582530 3075 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksdiuc7/build/release/bin/postgresql.jar:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksdiuc7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 21:51:39.582546 3075 mini_ranger_kms.cc:240] Using host: 127.3.0.212
I20260928 21:51:39.585566 3075 mini_ranger_kms.cc:276] Ranger KMS PID: 15256
I20260928 21:51:39.585634 3075 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.0.212:54515
Sep 28, 2026 9:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 21:51:41.592157 3075 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.636s user 0.008s sys 0.120s
I20260928 21:51:41.592262 3075 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 21:51:41.592305 3075 mini_ranger_kms.cc:401] 127.3.0.212:54515/kms/v1/keys
Sep 28 21:51:41 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632301, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.0.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 21:51:42 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632301, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.0.212@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
Sep 28 21:51:42 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632301, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.0.212@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
Sep 28 21:51:44 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:51:44.684880 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:45783
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:45783
--ranger_config_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:54515/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:44.763842 15307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:44.764050 15307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:44.764070 15307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:44.765225 15307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:51:44.765269 15307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:44.765282 15307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:44.765295 15307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:51:44.765319 15307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:51:44.766857 15307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.0.212:54515/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:45783
--ranger_config_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:45783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:44.767086 15307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:44.767277 15307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:44.772007 15312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:44.772070 15313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:44.772023 15307 server_base.cc:1034] running on GCE node
W20260928 21:51:44.772029 15315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:44.772528 15307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:44.772822 15307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:44.773975 15307 hybrid_clock.cc:648] HybridClock initialized: now 1790632304773942 us; error 33 us; skew 500 ppm
Sep 28 21:51:44 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632304, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:44.776234 15307 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:51:44.776695 15307 webserver.cc:460] Webserver started at http://127.3.0.254:38155/ using document root <none> and password file <none>
I20260928 21:51:44.776896 15307 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:44.776937 15307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:44.777064 15307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:51:44 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632304, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:44.837468 15307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "48b2aed6453a4310a5832f65081e38a9"
format_stamp: "Formatted at 2026-09-28 21:51:44 on dist-test-slave-jrck"
server_key: "431a377326111bce812902341cee8204"
server_key_iv: "f585646793ce51987f63070a4c18c581"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:44.837853 15307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "48b2aed6453a4310a5832f65081e38a9"
format_stamp: "Formatted at 2026-09-28 21:51:44 on dist-test-slave-jrck"
server_key: "431a377326111bce812902341cee8204"
server_key_iv: "f585646793ce51987f63070a4c18c581"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:44.839254 15307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:51:44.853961 15322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:44.854133 15307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:44.854230 15307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "48b2aed6453a4310a5832f65081e38a9"
format_stamp: "Formatted at 2026-09-28 21:51:44 on dist-test-slave-jrck"
server_key: "431a377326111bce812902341cee8204"
server_key_iv: "f585646793ce51987f63070a4c18c581"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:44.854333 15307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:44.883141 15307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:44.885015 15307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:44.885146 15307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:44.889426 15307 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:45783
I20260928 21:51:44.889495 15374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:45783 every 8 connection(s)
I20260928 21:51:44.889830 15307 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:51:44.890245 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 15307
I20260928 21:51:44.890350 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:51:44.890555 15375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:44.890570 3075 external_mini_cluster.cc:1428] Setting key 69301d590c3b31e4ab03281e36c4a82e
I20260928 21:51:44.892954 15375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: Bootstrap starting.
I20260928 21:51:44.893574 15375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:44.893886 15375 log.cc:824] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:44.894528 15375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: No bootstrap required, opened a new log
I20260928 21:51:44.896737 15375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } }
I20260928 21:51:44.896883 15375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:44.896914 15375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b2aed6453a4310a5832f65081e38a9, State: Initialized, Role: FOLLOWER
I20260928 21:51:44.897042 15375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } }
I20260928 21:51:44.897118 15375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:51:44.897150 15375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:51:44.897194 15375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:44.897794 15375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } }
I20260928 21:51:44.897881 15375 leader_election.cc:302] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48b2aed6453a4310a5832f65081e38a9; no voters:
I20260928 21:51:44.898046 15375 leader_election.cc:288] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:51:44.898100 15380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:51:44.898239 15380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 1 LEADER]: Becoming Leader. State: Replica: 48b2aed6453a4310a5832f65081e38a9, State: Running, Role: LEADER
I20260928 21:51:44.898316 15375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:51:44.898353 15380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } }
I20260928 21:51:44.898655 15380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48b2aed6453a4310a5832f65081e38a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } } }
I20260928 21:51:44.898723 15380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:44.898844 15380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48b2aed6453a4310a5832f65081e38a9. Latest consensus state: current_term: 1 leader_uuid: "48b2aed6453a4310a5832f65081e38a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } } }
I20260928 21:51:44.898885 15380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:44.898994 15386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:51:44.899444 15386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 21:51:44 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:44.899745 15375 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:51:44.903116 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:44.891088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.3.0.254:45783)
0928 21:51:44.892584 (+ 1496us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:44.892594 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:44.892628 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:51:44.893151 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:44.893162 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:44.893227 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:44.893397 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:44.894130 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:44.895537 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:44.896643 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:44.897307 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:44.897424 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:44.900159 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:44.900164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:44.900164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:44.900177 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:44.901717 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:44.902092 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:44.902095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:44.902096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:44.902170 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:44.902341 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:44.902342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:44.902343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:44.902567 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:44.902721 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:44.902811 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 21:51:44.902924 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1443,"thread_start_us":68,"threads_started":1}
I20260928 21:51:44.907323 15386 catalog_manager.cc:1347] Generated new cluster ID: 692f983b054a4d4aaa563fa7d482ef48
I20260928 21:51:44.907545 15386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:51:44.921640 15386 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:51:44.922379 15386 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:51:44.930287 15386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: Generated new TSK 0
I20260928 21:51:44.930506 15386 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:51:45.588773 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:54515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45783
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:45.673430 15420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:45.673651 15420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:45.673683 15420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:45.674847 15420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:45.674892 15420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:45.674927 15420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:51:45.676445 15420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.0.212:54515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:45.676728 15420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:45.676957 15420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:45.682886 15426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:45.682861 15425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:45.683135 15428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:45.683267 15420 server_base.cc:1034] running on GCE node
I20260928 21:51:45.683470 15420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:45.683714 15420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:45.684849 15420 hybrid_clock.cc:648] HybridClock initialized: now 1790632305684837 us; error 31 us; skew 500 ppm
Sep 28 21:51:45 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:45.687320 15420 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:51:45.687795 15420 webserver.cc:460] Webserver started at http://127.3.0.193:42229/ using document root <none> and password file <none>
I20260928 21:51:45.688021 15420 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:45.688071 15420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:45.688197 15420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:51:45 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:45.704219 15420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "e31241ea57fb49ffaec06b300c3c1ce1"
format_stamp: "Formatted at 2026-09-28 21:51:45 on dist-test-slave-jrck"
server_key: "070adaa6697f683284154700021ade09"
server_key_iv: "3f49c489c0d1b0c3a166947c26f4d3e2"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:45.704737 15420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "e31241ea57fb49ffaec06b300c3c1ce1"
format_stamp: "Formatted at 2026-09-28 21:51:45 on dist-test-slave-jrck"
server_key: "070adaa6697f683284154700021ade09"
server_key_iv: "3f49c489c0d1b0c3a166947c26f4d3e2"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:45.706745 15420 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:51:45.723716 15435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:45.723987 15420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260928 21:51:45.724092 15420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "e31241ea57fb49ffaec06b300c3c1ce1"
format_stamp: "Formatted at 2026-09-28 21:51:45 on dist-test-slave-jrck"
server_key: "070adaa6697f683284154700021ade09"
server_key_iv: "3f49c489c0d1b0c3a166947c26f4d3e2"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:45.724208 15420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:45.743741 15420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:45.746081 15420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:45.746248 15420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:45.746508 15420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:45.746896 15420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:45.746948 15420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:45.746978 15420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:45.746994 15420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:45.753687 15420 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:46751
I20260928 21:51:45.754191 15420 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:51:45.754765 15549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:46751 every 8 connection(s)
Sep 28 21:51:45 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:45.764329 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 15420
I20260928 21:51:45.764425 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:51:45.764635 3075 external_mini_cluster.cc:1428] Setting key 2d20f08c43554218ae3f6d2a2830f423
I20260928 21:51:45.765425 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:45.755387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:48193 (local address 127.3.0.254:45783)
0928 21:51:45.755630 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:45.755632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:45.755722 (+ 90us) server_negotiation.cc:407] Connection header received
0928 21:51:45.756321 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:45.756322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:45.756335 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:45.756381 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:45.757255 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:45.757870 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:45.759304 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:45.759618 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:45.759699 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:45.762243 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:45.762244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:45.762244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:45.762251 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:45.763754 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:45.764253 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:45.764255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:45.764255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:45.764333 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:45.764648 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:45.764649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:45.764650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:45.764803 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:45.764869 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:45.765256 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 21:51:45.765337 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":76,"threads_started":1}
I20260928 21:51:45.765918 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:45.755340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:45783 (local address 127.3.0.193:48193)
0928 21:51:45.755593 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:45.755638 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:45.756196 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:45.756598 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:45.756602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:45.756863 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:45.757145 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:45.757153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:45.757960 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:45.757961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:45.759191 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:45.759194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:45.759784 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:45.759785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:45.759924 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:45.760530 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:45.760540 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:45.762140 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:45.763830 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:45.763837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:45.763840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:45.764145 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:45.764409 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:45.764410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:45.764410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:45.764578 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:45.764899 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:45.764906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:45.765071 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 21:51:45.765516 (+ 445us) client_negotiation.cc:240] Negotiation successful
0928 21:51:45.765686 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260928 21:51:45.767231 15550 heartbeater.cc:347] Connected to a master server at 127.3.0.254:45783
I20260928 21:51:45.767334 15550 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:45.767522 15550 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
I20260928 21:51:45.768102 15339 ts_manager.cc:194] Registered new tserver with Master: e31241ea57fb49ffaec06b300c3c1ce1 (127.3.0.193:46751)
I20260928 21:51:45.769026 15339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:48193
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:51:45.790699 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:54515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45783
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:45.894485 15557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:45.894691 15557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:45.894726 15557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:45.895928 15557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:45.895972 15557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:45.896008 15557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:51:45.897536 15557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.0.212:54515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:45.897768 15557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:45.898025 15557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:45.902801 15565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:45.902838 15567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:45.903163 15564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:45.903494 15557 server_base.cc:1034] running on GCE node
I20260928 21:51:45.903709 15557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:45.903955 15557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:45.905097 15557 hybrid_clock.cc:648] HybridClock initialized: now 1790632305905079 us; error 33 us; skew 500 ppm
Sep 28 21:51:45 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:45.908084 15557 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:51:45.908587 15557 webserver.cc:460] Webserver started at http://127.3.0.194:38629/ using document root <none> and password file <none>
I20260928 21:51:45.908828 15557 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:45.908876 15557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:45.909009 15557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:51:45 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:45.921188 15557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "aa1f1536172b435e90ab51187f3fa7ea"
format_stamp: "Formatted at 2026-09-28 21:51:45 on dist-test-slave-jrck"
server_key: "a078362afd7a7243af887b9625778273"
server_key_iv: "9390b2b924909768db78ccb0e8cd16db"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:45.921718 15557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "aa1f1536172b435e90ab51187f3fa7ea"
format_stamp: "Formatted at 2026-09-28 21:51:45 on dist-test-slave-jrck"
server_key: "a078362afd7a7243af887b9625778273"
server_key_iv: "9390b2b924909768db78ccb0e8cd16db"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:45.928751 15557 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260928 21:51:45.944427 15574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:45.944715 15557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:51:45.944831 15557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "aa1f1536172b435e90ab51187f3fa7ea"
format_stamp: "Formatted at 2026-09-28 21:51:45 on dist-test-slave-jrck"
server_key: "a078362afd7a7243af887b9625778273"
server_key_iv: "9390b2b924909768db78ccb0e8cd16db"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:45.944945 15557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 21:51:45 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:45.962015 15557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:45.964783 15557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:45.965014 15557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:45.965343 15557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:45.965757 15557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:45.965855 15557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:45.965930 15557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:45.965994 15557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:45.972338 15557 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:46515
I20260928 21:51:45.974709 15687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:46515 every 8 connection(s)
I20260928 21:51:45.975258 15557 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:51:45.976850 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 15557
I20260928 21:51:45.976954 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:51:45.977154 3075 external_mini_cluster.cc:1428] Setting key 8a521c00d750586985a251bc0f5da859
Sep 28 21:51:45 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632305, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:45.987772 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:45.976558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:37831 (local address 127.3.0.254:45783)
0928 21:51:45.976609 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:45.976611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:45.977243 (+ 632us) server_negotiation.cc:407] Connection header received
0928 21:51:45.978370 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:45.978371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:45.978383 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:45.978411 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:45.979213 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:45.979960 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:45.981482 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:45.981757 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:45.981797 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:45.984400 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:45.984402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:45.984402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:45.984411 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:45.986109 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:45.986621 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:45.986623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:45.986623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:45.986682 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:45.986957 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:45.986958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:45.986959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:45.987099 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:45.987140 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:45.987576 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 21:51:45.987674 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:51:45.988713 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:45.976719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:45783 (local address 127.3.0.194:37831)
0928 21:51:45.977081 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:45.977128 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:45.978241 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:45.978593 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:45.978597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:45.978835 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:45.979108 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:45.979114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:45.980080 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:45.980081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:45.981383 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:45.981386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:45.981897 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:45.981898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:45.982042 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:45.982641 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:45.982651 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:45.984284 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:45.986205 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:45.986213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:45.986215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:45.986532 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:45.986750 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:45.986751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:45.986751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:45.986888 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:45.987184 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:45.987190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:45.987415 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 21:51:45.988223 (+ 808us) client_negotiation.cc:240] Negotiation successful
0928 21:51:45.988418 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":87,"threads_started":1}
I20260928 21:51:45.989408 15688 heartbeater.cc:347] Connected to a master server at 127.3.0.254:45783
I20260928 21:51:45.989502 15688 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:45.989703 15688 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
I20260928 21:51:45.990101 15339 ts_manager.cc:194] Registered new tserver with Master: aa1f1536172b435e90ab51187f3fa7ea (127.3.0.194:46515)
I20260928 21:51:45.990620 15339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:37831
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:51:46.021617 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:54515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45783
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:46.155028 15693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:46.155386 15693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:46.155469 15693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:46.157246 15693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:46.157337 15693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:46.157519 15693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:51:46.159698 15693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.0.212:54515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:45783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:46.160112 15693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:46.160421 15693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:46.167094 15699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:46.167124 15700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:46.167112 15702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:46.168413 15693 server_base.cc:1034] running on GCE node
I20260928 21:51:46.168875 15693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:46.169173 15693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:46.172487 15693 hybrid_clock.cc:648] HybridClock initialized: now 1790632306172433 us; error 37 us; skew 500 ppm
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:46.175616 15693 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:51:46.176126 15693 webserver.cc:460] Webserver started at http://127.3.0.195:40531/ using document root <none> and password file <none>
I20260928 21:51:46.176373 15693 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:46.176419 15693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:46.176564 15693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:46.190996 15693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "ad16ac0fc57f45b0a72323338e962833"
format_stamp: "Formatted at 2026-09-28 21:51:46 on dist-test-slave-jrck"
server_key: "0c7be6fa15b65ddcf59938e045106f89"
server_key_iv: "a0f5de571cc43fefb6f2c1d7089aa31e"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:46.191422 15693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "ad16ac0fc57f45b0a72323338e962833"
format_stamp: "Formatted at 2026-09-28 21:51:46 on dist-test-slave-jrck"
server_key: "0c7be6fa15b65ddcf59938e045106f89"
server_key_iv: "a0f5de571cc43fefb6f2c1d7089aa31e"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:46.194301 15693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260928 21:51:46.221676 15709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:46.221925 15693 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.001s
I20260928 21:51:46.222004 15693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "ad16ac0fc57f45b0a72323338e962833"
format_stamp: "Formatted at 2026-09-28 21:51:46 on dist-test-slave-jrck"
server_key: "0c7be6fa15b65ddcf59938e045106f89"
server_key_iv: "a0f5de571cc43fefb6f2c1d7089aa31e"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:46.222101 15693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:46.244151 15693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:46.246644 15693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:46.246852 15693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:46.247120 15693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:46.247519 15693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:46.247603 15693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:46.247668 15693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:46.247718 15693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:46.254493 15693 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:41281
I20260928 21:51:46.254973 15693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:51:46.256637 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 15693
I20260928 21:51:46.256726 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:51:46.256924 3075 external_mini_cluster.cc:1428] Setting key 2651ccd03f9c77f6dfb312ca6f3a45a3
I20260928 21:51:46.258535 15822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:41281 every 8 connection(s)
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:46.269510 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:46.259412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:43853 (local address 127.3.0.254:45783)
0928 21:51:46.259455 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:46.259458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:46.259731 (+ 273us) server_negotiation.cc:407] Connection header received
0928 21:51:46.260374 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:46.260375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:46.260388 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:46.260445 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:46.261204 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.261997 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.263410 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.263661 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.263728 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:46.266309 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:46.266312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:46.266312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:46.266320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:46.267908 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:46.268393 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:46.268394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:46.268395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:46.268478 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:46.268751 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:46.268752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:46.268752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:46.268890 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:46.268930 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:46.269320 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 21:51:46.269411 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:51:46.269659 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:46.259394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:45783 (local address 127.3.0.195:43853)
0928 21:51:46.259574 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:46.259629 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:46.260255 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:46.260557 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:46.260562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:46.260825 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:46.261109 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:46.261116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.262101 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.262102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:46.263338 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:46.263341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.263760 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.263761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:46.263904 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:46.264585 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:46.264595 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:46.266196 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:46.267996 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:46.268004 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:46.268007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:46.268302 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:46.268548 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:46.268548 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:46.268549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:46.268686 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:46.268970 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:46.268977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:46.269162 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 21:51:46.269272 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:51:46.269422 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260928 21:51:46.270243 15823 heartbeater.cc:347] Connected to a master server at 127.3.0.254:45783
I20260928 21:51:46.270349 15823 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:46.270532 15823 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
I20260928 21:51:46.270889 15339 ts_manager.cc:194] Registered new tserver with Master: ad16ac0fc57f45b0a72323338e962833 (127.3.0.195:41281)
I20260928 21:51:46.271368 15339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:43853
I20260928 21:51:46.279732 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:51:46.280441 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 15420
I20260928 21:51:46.286464 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 15557
I20260928 21:51:46.298151 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 15693
I20260928 21:51:46.304240 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 15307
I20260928 21:51:46.308838 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:45783
--webserver_interface=127.3.0.254
--webserver_port=38155
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:45783
--ranger_config_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:54515/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.0.212:54515/kms with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:46.389560 15826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:46.389796 15826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:46.389829 15826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:46.391041 15826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:51:46.391088 15826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:46.391116 15826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:46.391131 15826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:51:46.391146 15826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:51:46.392864 15826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.0.212:54515/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:45783
--ranger_config_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:45783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=38155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:46.393114 15826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:46.393321 15826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:46.398268 15831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:46.398413 15834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:46.398270 15832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:46.398474 15826 server_base.cc:1034] running on GCE node
I20260928 21:51:46.398772 15826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:46.399088 15826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:46.400252 15826 hybrid_clock.cc:648] HybridClock initialized: now 1790632306400239 us; error 32 us; skew 500 ppm
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:46.403226 15826 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:51:46.403820 15826 webserver.cc:460] Webserver started at http://127.3.0.254:38155/ using document root <none> and password file <none>
I20260928 21:51:46.404109 15826 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:46.404181 15826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:46.462833 15826 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:46.463506 15842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:46.463713 15826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:46.463835 15826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "48b2aed6453a4310a5832f65081e38a9"
format_stamp: "Formatted at 2026-09-28 21:51:44 on dist-test-slave-jrck"
server_key: "431a377326111bce812902341cee8204"
server_key_iv: "f585646793ce51987f63070a4c18c581"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:46.464311 15826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:46.477764 15826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:46.479347 15826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:46.479470 15826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:46.483700 15826 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:45783
I20260928 21:51:46.483740 15894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:45783 every 8 connection(s)
I20260928 21:51:46.484117 15826 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:51:46.485437 15895 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:51:46.486104 15895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: Bootstrap starting.
I20260928 21:51:46.493827 15895 log.cc:824] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:46.494251 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 15826
I20260928 21:51:46.494542 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:54515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:46751
--local_ip_for_outbound_sockets=127.3.0.193
--tserver_master_addrs=127.3.0.254:45783
--webserver_port=42229
--webserver_interface=127.3.0.193
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.0.212:54515/kms with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:51:46.495849 15895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:51:46.496094 15895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: Bootstrap complete.
I20260928 21:51:46.498402 15895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } }
I20260928 21:51:46.498644 15895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b2aed6453a4310a5832f65081e38a9, State: Initialized, Role: FOLLOWER
I20260928 21:51:46.498808 15895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } }
I20260928 21:51:46.498886 15895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:51:46.498930 15895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:51:46.498981 15895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:51:46.499989 15895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } }
I20260928 21:51:46.500089 15895 leader_election.cc:302] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48b2aed6453a4310a5832f65081e38a9; no voters:
I20260928 21:51:46.500250 15895 leader_election.cc:288] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:51:46.500308 15899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:51:46.500483 15899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [term 2 LEADER]: Becoming Leader. State: Replica: 48b2aed6453a4310a5832f65081e38a9, State: Running, Role: LEADER
I20260928 21:51:46.500576 15895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:51:46.500615 15899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } }
I20260928 21:51:46.500911 15900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "48b2aed6453a4310a5832f65081e38a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } } }
I20260928 21:51:46.500998 15900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:46.501247 15901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48b2aed6453a4310a5832f65081e38a9. Latest consensus state: current_term: 2 leader_uuid: "48b2aed6453a4310a5832f65081e38a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b2aed6453a4310a5832f65081e38a9" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 45783 } } }
I20260928 21:51:46.501309 15901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:46.501461 15905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:51:46.501994 15905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:51:46.502189 15905 catalog_manager.cc:1259] Loaded cluster ID: 692f983b054a4d4aaa563fa7d482ef48
I20260928 21:51:46.502278 15905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:51:46.502383 15895 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:51:46.503652 15905 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:51:46.503849 15905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: Loaded TSK: 0
I20260928 21:51:46.504077 15905 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:51:46.601492 15898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:46.601795 15898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:46.601851 15898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:46.603583 15898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:46.603657 15898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:46.603699 15898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:51:46.605991 15898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.0.212:54515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:46751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=42229
--tserver_master_addrs=127.3.0.254:45783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:46.606267 15898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:46.606547 15898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:46.614621 15933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:46.616720 15932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:46.616920 15935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:46.619554 15898 server_base.cc:1034] running on GCE node
I20260928 21:51:46.619936 15898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:46.620257 15898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:46.621441 15898 hybrid_clock.cc:648] HybridClock initialized: now 1790632306621427 us; error 45 us; skew 500 ppm
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:46.624821 15898 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:51:46.625398 15898 webserver.cc:460] Webserver started at http://127.3.0.193:42229/ using document root <none> and password file <none>
I20260928 21:51:46.625697 15898 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:46.625790 15898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:46.646044 15898 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:51:46.648713 15943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:46.648948 15898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260928 21:51:46.649045 15898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "e31241ea57fb49ffaec06b300c3c1ce1"
format_stamp: "Formatted at 2026-09-28 21:51:45 on dist-test-slave-jrck"
server_key: "070adaa6697f683284154700021ade09"
server_key_iv: "3f49c489c0d1b0c3a166947c26f4d3e2"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:46.649578 15898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:46.675264 15898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:46.680940 15898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:46.681075 15898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:46.681334 15898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:46.681741 15898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:46.681795 15898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:46.681825 15898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:46.681881 15898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:46.688107 15898 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:46751
I20260928 21:51:46.690294 16056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:46751 every 8 connection(s)
I20260928 21:51:46.690588 15898 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:46.700598 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 15898
I20260928 21:51:46.700917 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:54515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:46515
--local_ip_for_outbound_sockets=127.3.0.194
--tserver_master_addrs=127.3.0.254:45783
--webserver_port=38629
--webserver_interface=127.3.0.194
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.0.212:54515/kms with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:51:46.703999 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:46.691685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:56835 (local address 127.3.0.254:45783)
0928 21:51:46.691904 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:46.691915 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:46.692189 (+ 274us) server_negotiation.cc:407] Connection header received
0928 21:51:46.692968 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:46.692977 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:46.693053 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:46.693207 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:46.693926 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.695455 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.697013 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.697652 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.697756 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:46.700270 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:46.700275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:46.700276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:46.700287 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:46.702113 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:46.702598 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:46.702602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:46.702603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:46.702714 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:46.702985 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:46.702985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:46.702986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:46.703297 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:46.703379 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:46.703699 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 21:51:46.703808 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":81,"threads_started":1}
I20260928 21:51:46.704008 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:46.691583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:45783 (local address 127.3.0.193:56835)
0928 21:51:46.691995 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:46.692043 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:46.692702 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:46.693272 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:46.693277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:46.693506 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:46.693787 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:46.693793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.695573 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.695575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:46.696914 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:46.696917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.697871 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.697873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:46.698007 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:46.698544 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:46.698555 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:46.700165 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:46.702216 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:46.702223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:46.702226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:46.702508 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:46.702789 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:46.702790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:46.702790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:46.702914 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:46.703407 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:46.703412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:46.703535 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 21:51:46.703625 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 21:51:46.703780 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":86,"threads_started":1}
I20260928 21:51:46.716415 16057 heartbeater.cc:347] Connected to a master server at 127.3.0.254:45783
I20260928 21:51:46.716583 16057 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:46.716810 16057 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
W20260928 21:51:46.717748 16057 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:45783 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:51:46.717814 16057 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:46.717880 16057 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
I20260928 21:51:46.718163 16057 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:46.718240 16057 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
W20260928 21:51:46.718506 16057 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:51:46.822860 16061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:46.823167 16061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:46.823210 16061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:46.825054 16061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:46.825119 16061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:46.825171 16061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:51:46.827436 16061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.0.212:54515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:46515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=38629
--tserver_master_addrs=127.3.0.254:45783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:46.827680 16061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:46.827962 16061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:46.834268 16067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:46.834349 16069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:46.834869 16061 server_base.cc:1034] running on GCE node
W20260928 21:51:46.834262 16066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:46.836817 16061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:46.837129 16061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:46.838361 16061 hybrid_clock.cc:648] HybridClock initialized: now 1790632306838326 us; error 46 us; skew 500 ppm
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:46.844488 16061 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:51:46.844951 16061 webserver.cc:460] Webserver started at http://127.3.0.194:38629/ using document root <none> and password file <none>
I20260928 21:51:46.845194 16061 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:46.845258 16061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:46.864266 16061 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 21:51:46.865306 16077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:46.865532 16061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:51:46.865821 16061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "aa1f1536172b435e90ab51187f3fa7ea"
format_stamp: "Formatted at 2026-09-28 21:51:45 on dist-test-slave-jrck"
server_key: "a078362afd7a7243af887b9625778273"
server_key_iv: "9390b2b924909768db78ccb0e8cd16db"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:46.866294 16061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:46.884384 16061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:46.886588 16061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:46.886770 16061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:46.887004 16061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:46.887395 16061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:46.887449 16061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:46.887487 16061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:46.887512 16061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:46.894743 16061 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:46515
I20260928 21:51:46.894939 16190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:46515 every 8 connection(s)
I20260928 21:51:46.895290 16061 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:51:46.900171 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 16061
I20260928 21:51:46.900493 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.0.212:54515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:41281
--local_ip_for_outbound_sockets=127.3.0.195
--tserver_master_addrs=127.3.0.254:45783
--webserver_port=40531
--webserver_interface=127.3.0.195
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.0.212:54515/kms with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:51:46 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632306, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:46.906939 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:46.896091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:40809 (local address 127.3.0.254:45783)
0928 21:51:46.896152 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:46.896154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:46.896494 (+ 340us) server_negotiation.cc:407] Connection header received
0928 21:51:46.897251 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:46.897253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:46.897274 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:46.897344 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:46.898055 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.898865 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.900522 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.900795 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.900843 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:46.903690 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:46.903692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:46.903693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:46.903701 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:46.905319 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:46.905814 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:46.905816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:46.905816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:46.905877 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:46.906166 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:46.906167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:46.906168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:46.906312 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:46.906358 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:46.906750 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 21:51:46.906842 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:51:46.907449 16193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:46.896108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:45783 (local address 127.3.0.194:40809)
0928 21:51:46.896314 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:46.896363 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:46.897120 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:46.897415 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:46.897419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:46.897659 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:46.897943 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:46.897950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.898975 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.898976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:46.900350 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:46.900353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:46.901182 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:46.901183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:46.901398 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:46.901949 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:46.901960 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:46.903579 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:46.905415 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:46.905421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:46.905423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:46.905711 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:46.905946 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:46.905946 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:46.905947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:46.906077 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:46.906408 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:46.906414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:46.906580 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 21:51:46.907036 (+ 456us) client_negotiation.cc:240] Negotiation successful
0928 21:51:46.907222 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260928 21:51:46.908004 16191 heartbeater.cc:347] Connected to a master server at 127.3.0.254:45783
I20260928 21:51:46.908099 16191 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:46.908285 16191 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
W20260928 21:51:46.908962 16191 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:45783 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:51:46.909078 16191 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:46.909137 16191 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
I20260928 21:51:46.909539 16191 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:46.909618 16191 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
W20260928 21:51:46.909962 16191 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:51:47.002496 16194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:47.002684 16194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:47.002707 16194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:47.004207 16194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:47.004271 16194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:47.004309 16194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:51:47.006222 16194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.0.212:54515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:41281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=40531
--tserver_master_addrs=127.3.0.254:45783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:47.006469 16194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:47.006711 16194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:47.011950 16201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:47.011927 16203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:47.012504 16194 server_base.cc:1034] running on GCE node
W20260928 21:51:47.015937 16200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:47.016196 16194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:47.016472 16194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:47.017628 16194 hybrid_clock.cc:648] HybridClock initialized: now 1790632307017612 us; error 34 us; skew 500 ppm
Sep 28 21:51:47 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:47.020663 16194 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:51:47.021207 16194 webserver.cc:460] Webserver started at http://127.3.0.195:40531/ using document root <none> and password file <none>
I20260928 21:51:47.021476 16194 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:47.021574 16194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:51:47 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:47.034049 16194 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:51:47.034940 16211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:47.035125 16194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:47.035212 16194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "ad16ac0fc57f45b0a72323338e962833"
format_stamp: "Formatted at 2026-09-28 21:51:46 on dist-test-slave-jrck"
server_key: "0c7be6fa15b65ddcf59938e045106f89"
server_key_iv: "a0f5de571cc43fefb6f2c1d7089aa31e"
server_key_version: "kuduclusterkey@0"
I20260928 21:51:47.035550 16194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:47.058288 16194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:47.060232 16194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:47.060372 16194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:47.060663 16194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:47.061007 16194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:47.061057 16194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:47.061087 16194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:47.061115 16194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:47.067346 16194 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:41281
I20260928 21:51:47.067696 16324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:41281 every 8 connection(s)
I20260928 21:51:47.067799 16194 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:51:47.073025 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 16194
Sep 28 21:51:47 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:47.081569 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:47.068927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:54493 (local address 127.3.0.254:45783)
0928 21:51:47.068997 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:47.068999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:47.072676 (+ 3677us) server_negotiation.cc:407] Connection header received
0928 21:51:47.073376 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:47.073377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:47.073390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:47.073418 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:47.074328 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:47.074856 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:47.076072 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:47.076256 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:47.076309 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:47.078602 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:47.078604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:47.078605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:47.078612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:47.080121 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:47.080625 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:47.080626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:47.080626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:47.080680 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:47.080911 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:47.080912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:47.080912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:47.081049 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:47.081121 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:47.081417 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 21:51:47.081502 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:51:47.082137 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:47.068633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:45783 (local address 127.3.0.195:54493)
0928 21:51:47.072519 (+ 3886us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:47.072576 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:47.073252 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:47.073767 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:47.073772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:47.073999 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:47.074245 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:47.074250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:47.074939 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:47.074940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:47.075999 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:47.076001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:47.076339 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:47.076340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:47.076448 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:47.077064 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:47.077075 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:47.078499 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:47.080219 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:47.080224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:47.080226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:47.080543 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:47.080733 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:47.080734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:47.080734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:47.080845 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:47.081127 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:47.081131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:47.081256 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 21:51:47.081694 (+ 438us) client_negotiation.cc:240] Negotiation successful
0928 21:51:47.081856 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260928 21:51:47.082592 16325 heartbeater.cc:347] Connected to a master server at 127.3.0.254:45783
I20260928 21:51:47.082698 16325 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:47.082894 16325 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
W20260928 21:51:47.083534 16325 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:45783 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:51:47.083614 16325 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:47.083688 16325 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
I20260928 21:51:47.083995 16325 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:47.084066 16325 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
W20260928 21:51:47.084353 16325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:51:47.084547 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:47.073511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.3.0.254:45783)
0928 21:51:47.074365 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:47.074366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:47.074373 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:51:47.074389 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:47.074389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:47.074398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:47.074453 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:47.075649 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:47.076427 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:47.077488 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:47.077786 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:47.077861 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:47.079070 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:47.079071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:47.079071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:47.079078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:47.081770 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:47.082329 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:47.082331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:47.082331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:47.082392 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:47.083088 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:47.083089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:47.083090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:47.083233 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:47.083857 (+ 624us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:47.083903 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 21:51:47.084213 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":64,"threads_started":1}
Sep 28 21:51:47 dist-test-slave-jrck krb5kdc[14863](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790632307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:47.718693 16057 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:47.718863 16057 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
I20260928 21:51:47.719296 15859 ts_manager.cc:194] Registered new tserver with Master: e31241ea57fb49ffaec06b300c3c1ce1 (127.3.0.193:46751)
I20260928 21:51:47.720309 15859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:56835
I20260928 21:51:47.910146 16191 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:47.910305 16191 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
I20260928 21:51:47.910712 15859 ts_manager.cc:194] Registered new tserver with Master: aa1f1536172b435e90ab51187f3fa7ea (127.3.0.194:46515)
I20260928 21:51:47.911259 15859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:40809
Sep 28 21:51:47 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
I20260928 21:51:48.084542 16325 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:48.084702 16325 heartbeater.cc:511] Master 127.3.0.254:45783 requested a full tablet report, sending...
I20260928 21:51:48.084995 15859 ts_manager.cc:194] Registered new tserver with Master: ad16ac0fc57f45b0a72323338e962833 (127.3.0.195:41281)
I20260928 21:51:48.085533 15859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:54493
I20260928 21:51:48.091542 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:51:48.100705 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.092803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.3.0.254:45783)
0928 21:51:48.093180 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:48.093182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:48.093190 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:51:48.093211 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:48.093211 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:48.093222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:48.093923 (+ 701us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:48.093961 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.094675 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.095737 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.096098 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.096511 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.097660 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:48.097661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:48.097662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:48.097669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:48.099396 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:48.099808 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:48.099809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:48.099809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:48.099870 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:48.100053 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:48.100053 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:48.100054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:48.100182 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:48.100294 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:48.100511 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 21:51:48.100596 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":65,"threads_started":1}
I20260928 21:51:48.102998 15859 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:51:48.105504 15859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:51:48.114831 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.108661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:46751 (local address 127.0.0.1:35304)
0928 21:51:48.109106 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:48.109498 (+ 392us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:48.109564 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:48.109926 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:48.109930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:48.109932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:48.110012 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:48.110014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.111737 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.111739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:48.113098 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:48.113101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.114497 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.114498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:48.114589 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.114604 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:51:48.114674 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:51:48.114733 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":91,"threads_started":1}
I20260928 21:51:48.115489 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.108944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.3.0.193:46751)
0928 21:51:48.109693 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:48.109699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:48.109708 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:51:48.109743 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:48.109745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:48.109759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:48.109827 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:48.110088 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.111655 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.113194 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.114325 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.114394 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.114460 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:48.115363 (+ 903us) server_negotiation.cc:299] Negotiation successful
0928 21:51:48.115410 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":62,"threads_started":1}
I20260928 21:51:48.116258 15991 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed16ab0227444bd9c555dafa99499c0 (DEFAULT_TABLE table=test-table [id=310ca1a7d76f4784bb848d30ebd18c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:48.116683 15991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed16ab0227444bd9c555dafa99499c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:48.116854 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.108678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:41281 (local address 127.0.0.1:33322)
0928 21:51:48.109481 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:48.109513 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:48.109569 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:48.112892 (+ 3323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:48.112893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:48.112897 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:48.112943 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:48.112944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.114186 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.114187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:48.115309 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:48.115312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.116612 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.116613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:48.116676 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.116681 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:51:48.116737 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:51:48.116782 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":85,"threads_started":1}
I20260928 21:51:48.117164 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.109187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.3.0.195:41281)
0928 21:51:48.112626 (+ 3439us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:48.112631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:48.112642 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:51:48.112678 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:48.112680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:48.112697 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:48.112786 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:48.113019 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.114088 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.115412 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.116189 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.116265 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.116319 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:48.116837 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 21:51:48.116886 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3395,"thread_start_us":60,"threads_started":1}
I20260928 21:51:48.117852 16250 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed16ab0227444bd9c555dafa99499c0 (DEFAULT_TABLE table=test-table [id=310ca1a7d76f4784bb848d30ebd18c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:48.118212 16250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed16ab0227444bd9c555dafa99499c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:48.120568 16358 tablet_bootstrap.cc:492] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1: Bootstrap starting.
I20260928 21:51:48.121207 16358 tablet_bootstrap.cc:654] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:48.121912 16358 log.cc:824] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:48.122215 16359 tablet_bootstrap.cc:492] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: Bootstrap starting.
I20260928 21:51:48.122862 16359 tablet_bootstrap.cc:654] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:48.123207 16359 log.cc:824] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:48.123971 16358 tablet_bootstrap.cc:492] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1: No bootstrap required, opened a new log
I20260928 21:51:48.124083 16358 ts_tablet_manager.cc:1397] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:51:48.126358 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.109334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49626 (local address 127.3.0.194:46515)
0928 21:51:48.117565 (+ 8231us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:48.117570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:48.117581 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:51:48.117615 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:48.117617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:48.117633 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:48.117718 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:48.122334 (+ 4616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.123437 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.124705 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.125454 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.125523 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.125581 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:48.126224 (+ 643us) server_negotiation.cc:299] Negotiation successful
0928 21:51:48.126284 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8193,"thread_start_us":52,"threads_started":1}
I20260928 21:51:48.125840 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.108920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:46515 (local address 127.0.0.1:49626)
0928 21:51:48.115240 (+ 6320us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:48.115250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:48.115293 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:48.122170 (+ 6877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:48.122171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:48.122173 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:48.122217 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:48.122219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.123539 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.123540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:48.124606 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:48.124609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.125615 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.125616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:48.125678 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.125682 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:51:48.125718 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 21:51:48.125764 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6306,"thread_start_us":72,"threads_started":1}
I20260928 21:51:48.126272 16358 raft_consensus.cc:348] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 46751 } } peers { permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 } } peers { permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 } }
I20260928 21:51:48.126737 16358 raft_consensus.cc:374] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:48.126808 16358 raft_consensus.cc:729] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31241ea57fb49ffaec06b300c3c1ce1, State: Initialized, Role: FOLLOWER
I20260928 21:51:48.126961 16358 consensus_queue.cc:260] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 46751 } } peers { permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 } } peers { permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 } }
I20260928 21:51:48.127223 16358 ts_tablet_manager.cc:1428] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:51:48.127223 16125 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed16ab0227444bd9c555dafa99499c0 (DEFAULT_TABLE table=test-table [id=310ca1a7d76f4784bb848d30ebd18c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:48.127347 16057 heartbeater.cc:503] Master 127.3.0.254:45783 was elected leader, sending a full tablet report...
I20260928 21:51:48.127557 16125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed16ab0227444bd9c555dafa99499c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:48.127925 16359 tablet_bootstrap.cc:492] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: No bootstrap required, opened a new log
I20260928 21:51:48.127990 16359 ts_tablet_manager.cc:1397] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 21:51:48.129967 16359 raft_consensus.cc:348] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 46751 } } peers { permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 } } peers { permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 } }
I20260928 21:51:48.130086 16359 raft_consensus.cc:374] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:48.130105 16359 raft_consensus.cc:729] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad16ac0fc57f45b0a72323338e962833, State: Initialized, Role: FOLLOWER
I20260928 21:51:48.130199 16359 consensus_queue.cc:260] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 46751 } } peers { permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 } } peers { permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 } }
I20260928 21:51:48.130441 16359 ts_tablet_manager.cc:1428] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:51:48.130463 16325 heartbeater.cc:503] Master 127.3.0.254:45783 was elected leader, sending a full tablet report...
I20260928 21:51:48.131604 16364 tablet_bootstrap.cc:492] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: Bootstrap starting.
I20260928 21:51:48.132669 16364 tablet_bootstrap.cc:654] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:48.133039 16364 log.cc:824] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:48.133759 16364 tablet_bootstrap.cc:492] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: No bootstrap required, opened a new log
I20260928 21:51:48.133826 16364 ts_tablet_manager.cc:1397] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:51:48.136878 16364 raft_consensus.cc:348] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 46751 } } peers { permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 } } peers { permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 } }
I20260928 21:51:48.137034 16364 raft_consensus.cc:374] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:48.137063 16364 raft_consensus.cc:729] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1f1536172b435e90ab51187f3fa7ea, State: Initialized, Role: FOLLOWER
I20260928 21:51:48.137173 16364 consensus_queue.cc:260] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 46751 } } peers { permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 } } peers { permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 } }
I20260928 21:51:48.137835 16364 ts_tablet_manager.cc:1428] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 21:51:48.137972 16191 heartbeater.cc:503] Master 127.3.0.254:45783 was elected leader, sending a full tablet report...
W20260928 21:51:48.146636 16192 tablet.cc:2369] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 21:51:48 dist-test-slave-jrck krb5kdc[14863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632307, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for HTTP/127.3.0.212@KRBTEST.COM
W20260928 21:51:48.191918 16058 tablet.cc:2369] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:51:48.319339 16326 tablet.cc:2369] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:51:48.372836 16362 raft_consensus.cc:482] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:51:48.372984 16362 raft_consensus.cc:504] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 46751 } } peers { permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 } } peers { permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 } }
I20260928 21:51:48.373463 16362 leader_election.cc:288] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad16ac0fc57f45b0a72323338e962833 (127.3.0.195:41281), aa1f1536172b435e90ab51187f3fa7ea (127.3.0.194:46515)
I20260928 21:51:48.384907 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.374010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:42393 (local address 127.3.0.195:41281)
0928 21:51:48.374054 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:48.374057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:48.381199 (+ 7142us) server_negotiation.cc:407] Connection header received
0928 21:51:48.381316 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:48.381318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:48.381331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:48.381361 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:48.381603 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.382355 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.383899 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.384512 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.384559 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.384571 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:48.384792 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 21:51:48.384832 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:51:48.385390 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.373631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:41281 (local address 127.3.0.193:42393)
0928 21:51:48.381089 (+ 7458us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:48.381106 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:48.381238 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:48.381442 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:48.381443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:48.381444 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:48.381527 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:48.381529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.382473 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.382474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:48.383784 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:48.383788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.384644 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.384645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:48.384717 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.384721 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:51:48.385237 (+ 516us) client_negotiation.cc:240] Negotiation successful
0928 21:51:48.385287 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7423,"thread_start_us":106,"threads_started":1}
I20260928 21:51:48.385878 16279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed16ab0227444bd9c555dafa99499c0" candidate_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad16ac0fc57f45b0a72323338e962833" is_pre_election: true
I20260928 21:51:48.386053 16279 raft_consensus.cc:2454] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31241ea57fb49ffaec06b300c3c1ce1 in term 0.
I20260928 21:51:48.386409 15946 leader_election.cc:302] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad16ac0fc57f45b0a72323338e962833, e31241ea57fb49ffaec06b300c3c1ce1; no voters:
I20260928 21:51:48.386543 16362 raft_consensus.cc:2781] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:51:48.386584 16362 raft_consensus.cc:482] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:51:48.386605 16362 raft_consensus.cc:3037] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:48.387508 16362 raft_consensus.cc:504] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 46751 } } peers { permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 } } peers { permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 } }
I20260928 21:51:48.387651 16362 leader_election.cc:288] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [CANDIDATE]: Term 1 election: Requested vote from peers ad16ac0fc57f45b0a72323338e962833 (127.3.0.195:41281), aa1f1536172b435e90ab51187f3fa7ea (127.3.0.194:46515)
I20260928 21:51:48.387789 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.373996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:57265 (local address 127.3.0.194:46515)
0928 21:51:48.374042 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:48.374044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:48.384622 (+ 10578us) server_negotiation.cc:407] Connection header received
0928 21:51:48.384726 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:48.384727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:48.384741 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:48.384770 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:48.384975 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.385627 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.386882 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.387412 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.387460 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.387471 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:48.387677 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 21:51:48.387725 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:51:48.388013 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.373739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:46515 (local address 127.3.0.193:57265)
0928 21:51:48.384552 (+ 10813us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:48.384564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:48.384654 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:48.384828 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:48.384828 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:48.384830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:48.384888 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:48.384889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.385722 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.385723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:48.386783 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:48.386785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.387529 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.387530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:48.387597 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.387600 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:51:48.387891 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 21:51:48.387942 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10788,"thread_start_us":77,"threads_started":1}
I20260928 21:51:48.388072 16279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed16ab0227444bd9c555dafa99499c0" candidate_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad16ac0fc57f45b0a72323338e962833"
I20260928 21:51:48.388144 16279 raft_consensus.cc:3037] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:48.389029 16279 raft_consensus.cc:2454] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31241ea57fb49ffaec06b300c3c1ce1 in term 1.
I20260928 21:51:48.389434 16145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed16ab0227444bd9c555dafa99499c0" candidate_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1f1536172b435e90ab51187f3fa7ea" is_pre_election: true
I20260928 21:51:48.389582 16145 raft_consensus.cc:2454] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31241ea57fb49ffaec06b300c3c1ce1 in term 0.
I20260928 21:51:48.389685 16144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed16ab0227444bd9c555dafa99499c0" candidate_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1f1536172b435e90ab51187f3fa7ea"
I20260928 21:51:48.389731 16144 raft_consensus.cc:3037] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:48.391129 16144 raft_consensus.cc:2454] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31241ea57fb49ffaec06b300c3c1ce1 in term 1.
I20260928 21:51:48.391458 15947 leader_election.cc:302] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1f1536172b435e90ab51187f3fa7ea, e31241ea57fb49ffaec06b300c3c1ce1; no voters:
I20260928 21:51:48.391925 16362 raft_consensus.cc:2781] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:51:48.392095 16362 raft_consensus.cc:686] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 1 LEADER]: Becoming Leader. State: Replica: e31241ea57fb49ffaec06b300c3c1ce1, State: Running, Role: LEADER
I20260928 21:51:48.392186 16362 consensus_queue.cc:237] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 46751 } } peers { permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 } } peers { permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 } }
I20260928 21:51:48.393294 15859 catalog_manager.cc:5495] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e31241ea57fb49ffaec06b300c3c1ce1 (127.3.0.193). New cstate: current_term: 1 leader_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31241ea57fb49ffaec06b300c3c1ce1" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 46751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 } health_report { overall_health: UNKNOWN } } }
I20260928 21:51:48.468487 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.465926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.3.0.193:46751)
0928 21:51:48.465989 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:48.465992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:48.466001 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:51:48.466044 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:48.466045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:48.466069 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:48.466140 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:51:48.466229 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.466953 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.467676 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:48.467954 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:48.468017 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:48.468121 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:51:48.468258 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:51:48.468350 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:48.468365 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 21:51:48.468385 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:51:48.470252 16279 raft_consensus.cc:1261] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833 [term 1 FOLLOWER]: Refusing update from remote peer e31241ea57fb49ffaec06b300c3c1ce1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:51:48.470342 16144 raft_consensus.cc:1261] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea [term 1 FOLLOWER]: Refusing update from remote peer e31241ea57fb49ffaec06b300c3c1ce1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:51:48.470602 16371 consensus_queue.cc:1035] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad16ac0fc57f45b0a72323338e962833" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:51:48.470727 16362 consensus_queue.cc:1035] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa1f1536172b435e90ab51187f3fa7ea" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:51:48.473235 16381 mvcc.cc:205] Tried to move back new op lower bound from 7334429935491686400 to 7334429935175151616. Current Snapshot: MvccSnapshot[applied={T|T < 7334429935491686400}]
I20260928 21:51:48.474442 16384 mvcc.cc:205] Tried to move back new op lower bound from 7334429935491686400 to 7334429935175151616. Current Snapshot: MvccSnapshot[applied={T|T < 7334429935491686400}]
I20260928 21:51:48.476337 15859 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:51:48.476449 15859 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:51:48.477214 15859 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48b2aed6453a4310a5832f65081e38a9: Sending DeleteTablet for 3 replicas of tablet 3ed16ab0227444bd9c555dafa99499c0
I20260928 21:51:48.477510 15991 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ed16ab0227444bd9c555dafa99499c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:48 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:35304
I20260928 21:51:48.477545 16250 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ed16ab0227444bd9c555dafa99499c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:48 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:33322
I20260928 21:51:48.477756 16390 tablet_replica.cc:330] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1: stopping tablet replica
I20260928 21:51:48.477890 16125 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ed16ab0227444bd9c555dafa99499c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:48 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:49626
I20260928 21:51:48.477867 16390 raft_consensus.cc:2229] T 3ed16ab0227444bd9c555dafa99499c0 P e31241ea57fb49ffaec06b300c3c1ce1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:51:48.478091 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 15898
I20260928 21:51:48.478608 16392 tablet_replica.cc:330] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: stopping tablet replica
I20260928 21:51:48.478720 16392 raft_consensus.cc:2229] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:51:48.478776 16392 raft_consensus.cc:2258] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:51:48.478942 16391 tablet_replica.cc:330] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: stopping tablet replica
I20260928 21:51:48.479035 16391 raft_consensus.cc:2229] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:51:48.479321 16391 raft_consensus.cc:2258] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:51:48.479739 16391 ts_tablet_manager.cc:1905] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:51:48.481527 16391 ts_tablet_manager.cc:1918] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:51:48.481601 16391 log.cc:1196] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/3ed16ab0227444bd9c555dafa99499c0
I20260928 21:51:48.481837 16391 ts_tablet_manager.cc:1939] T 3ed16ab0227444bd9c555dafa99499c0 P ad16ac0fc57f45b0a72323338e962833: Deleting consensus metadata
I20260928 21:51:48.482209 15845 catalog_manager.cc:4841] TS ad16ac0fc57f45b0a72323338e962833 (127.3.0.195:41281): tablet 3ed16ab0227444bd9c555dafa99499c0 (table test-table [id=310ca1a7d76f4784bb848d30ebd18c09]) successfully deleted
I20260928 21:51:48.482537 16392 ts_tablet_manager.cc:1905] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:51:48.483587 15844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:46751 (error 108)
I20260928 21:51:48.484055 16392 ts_tablet_manager.cc:1918] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:51:48.484127 16392 log.cc:1196] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/3ed16ab0227444bd9c555dafa99499c0
I20260928 21:51:48.484400 16392 ts_tablet_manager.cc:1939] T 3ed16ab0227444bd9c555dafa99499c0 P aa1f1536172b435e90ab51187f3fa7ea: Deleting consensus metadata
I20260928 21:51:48.484974 15846 catalog_manager.cc:4841] TS aa1f1536172b435e90ab51187f3fa7ea (127.3.0.194:46515): tablet 3ed16ab0227444bd9c555dafa99499c0 (table test-table [id=310ca1a7d76f4784bb848d30ebd18c09]) successfully deleted
I20260928 21:51:48.485342 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 16061
I20260928 21:51:48.486181 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:48.485503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:46751 (local address 127.0.0.1:35322)
0928 21:51:48.486117 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:48.486156 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:46751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594}
W20260928 21:51:48.486310 15844 catalog_manager.cc:4584] TS e31241ea57fb49ffaec06b300c3c1ce1 (127.3.0.193:46751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ed16ab0227444bd9c555dafa99499c0: Network error: Client connection negotiation failed: client connection to 127.3.0.193:46751: connect: Connection refused (error 111)
I20260928 21:51:48.490315 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 16194
I20260928 21:51:48.495289 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 15826
I20260928 21:51:48.501520 3075 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 21:51:48.835039 3075 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 21:51:48.835124 3075 mini_ranger.cc:66] Stopping Ranger...
I20260928 21:51:49.169263 3075 mini_ranger.cc:68] Stopped Ranger
2026-09-28 21:51:49.169 UTC [14893] LOG: received smart shutdown request
2026-09-28 21:51:49.172 UTC [14893] LOG: background worker "logical replication launcher" (PID 14902) exited with exit code 1
2026-09-28 21:51:49.172 UTC [14897] LOG: shutting down
2026-09-28 21:51:49.312 UTC [14893] LOG: database system is shut down
2026-09-28T21:51:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46160 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:51:49 dist-test-slave-jrck krb5kdc[16402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:51:49 dist-test-slave-jrck krb5kdc[16402](info): set up 2 sockets
Sep 28 21:51:49 dist-test-slave-jrck krb5kdc[16402](info): commencing operation
krb5kdc: starting...
W20260928 21:51:50.352077 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:51:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:51:50.462978 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:39985
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:39985,127.3.0.253:41289,127.3.0.252:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:50.542122 16415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:50.542315 16415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:50.542335 16415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:50.543443 16415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:51:50.543489 16415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:50.543504 16415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:50.543515 16415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:51:50.543527 16415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:51:50.545094 16415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:39985,127.3.0.253:41289,127.3.0.252:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:39985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:50.545321 16415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:50.545526 16415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:50.550076 16421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:50.550056 16423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:50.550056 16420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:50.550326 16415 server_base.cc:1034] running on GCE node
I20260928 21:51:50.550566 16415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:50.550863 16415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:50.552017 16415 hybrid_clock.cc:648] HybridClock initialized: now 1790632310551981 us; error 33 us; skew 500 ppm
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:50.554411 16415 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:51:50.554824 16415 webserver.cc:460] Webserver started at http://127.3.0.254:33751/ using document root <none> and password file <none>
I20260928 21:51:50.555048 16415 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:50.555095 16415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:50.555202 16415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:51:50.556248 16415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "556cbc69eea4482ca523179af8f0275b"
format_stamp: "Formatted at 2026-09-28 21:51:50 on dist-test-slave-jrck"
server_key: "3eab75fd890824f2f767b45b3be96e1f"
server_key_iv: "c6330aac17400f5e1b827498980bda39"
server_key_version: "encryptionkey@0"
I20260928 21:51:50.556604 16415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "556cbc69eea4482ca523179af8f0275b"
format_stamp: "Formatted at 2026-09-28 21:51:50 on dist-test-slave-jrck"
server_key: "3eab75fd890824f2f767b45b3be96e1f"
server_key_iv: "c6330aac17400f5e1b827498980bda39"
server_key_version: "encryptionkey@0"
I20260928 21:51:50.557792 16415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:50.558468 16430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:50.558632 16415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:50.558712 16415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "556cbc69eea4482ca523179af8f0275b"
format_stamp: "Formatted at 2026-09-28 21:51:50 on dist-test-slave-jrck"
server_key: "3eab75fd890824f2f767b45b3be96e1f"
server_key_iv: "c6330aac17400f5e1b827498980bda39"
server_key_version: "encryptionkey@0"
I20260928 21:51:50.558766 16415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:50.567824 16415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:50.569839 16415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:50.569945 16415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:50.573678 16415 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:39985
I20260928 21:51:50.573707 16482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:39985 every 8 connection(s)
I20260928 21:51:50.574025 16415 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:51:50.574855 16483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:50.576360 16483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:51:50.577194 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 16415
I20260928 21:51:50.577306 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:51:50.577487 3075 external_mini_cluster.cc:1428] Setting key 14815fd7a3220ed8dd4d9e7111c34435
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.0.253@KRBTEST.COM; defaulting to no policy
I20260928 21:51:50.586333 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.577035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:39985 (local address 127.0.0.1:48288)
0928 21:51:50.577195 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.577243 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:50.577838 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:50.578114 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:50.578116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:50.578445 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:50.578667 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.578673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.579656 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.579657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.580412 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.580414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.581107 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.581108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.581167 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.581666 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:50.581675 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:50.583216 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:50.585045 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.585052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.585055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.585240 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.585458 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.585459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.585459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.585543 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.585856 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:50.585860 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:50.585937 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 21:51:50.586020 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 21:51:50.586196 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":57,"thread_start_us":68,"threads_started":1}
I20260928 21:51:50.586341 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.577189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.3.0.254:39985)
0928 21:51:50.577396 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.577401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.577437 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:51:50.577991 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.577993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.578010 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.578087 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.578773 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.579560 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.580529 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.581019 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.581128 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.583301 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.583309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.583310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.583317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.584965 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.585318 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.585321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.585321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.585406 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.585579 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.585580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.585580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.585799 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.585864 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.586042 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.586133 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":46,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.253@KRBTEST.COM" created.
I20260928 21:51:50.587183 16483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:51:50.587525 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.587406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:41289 (local address 127.0.0.1:54054)
0928 21:51:50.587453 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.587499 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 21:51:50.587661 16432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.253:41289: connect: Connection refused (error 111)
I20260928 21:51:50.588060 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.587980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:41289 (local address 127.0.0.1:54060)
0928 21:51:50.588019 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.588043 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 21:51:50.588155 16483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.253:41289: Network error: Client connection negotiation failed: client connection to 127.3.0.253:41289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab.
Entry for principal kudu/127.3.0.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab.
I20260928 21:51:50.600003 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.253:41289
--webserver_interface=127.3.0.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab
--principal=kudu/127.3.0.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:39985,127.3.0.253:41289,127.3.0.252:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:51:50.637378 16483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } attempt: 1
I20260928 21:51:50.637892 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.637742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:41289 (local address 127.0.0.1:54064)
0928 21:51:50.637807 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.637856 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 21:51:50.638209 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.638129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:41289 (local address 127.0.0.1:54072)
0928 21:51:50.638163 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.638189 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 21:51:50.638316 16483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.253:41289: Network error: Client connection negotiation failed: client connection to 127.3.0.253:41289: connect: Connection refused (error 111)
W20260928 21:51:50.679103 16490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:50.679329 16490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:50.679352 16490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:50.680533 16490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:51:50.680574 16490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:50.680586 16490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:50.680598 16490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:51:50.680610 16490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:51:50.682148 16490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:39985,127.3.0.253:41289,127.3.0.252:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab
--principal=kudu/127.3.0.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.253:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.0.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:50.682372 16490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:50.682577 16490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:50.687119 16495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:50.687153 16496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:50.687181 16490 server_base.cc:1034] running on GCE node
W20260928 21:51:50.687119 16498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:50.687561 16490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:50.687835 16490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:50.688992 16490 hybrid_clock.cc:648] HybridClock initialized: now 1790632310688976 us; error 27 us; skew 500 ppm
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:50.691354 16490 init.cc:377] Logged in from keytab as kudu/127.3.0.253@KRBTEST.COM (short username kudu)
I20260928 21:51:50.691756 16490 webserver.cc:460] Webserver started at http://127.3.0.253:45765/ using document root <none> and password file <none>
I20260928 21:51:50.691969 16490 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:50.692018 16490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:50.692139 16490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:51:50.693204 16490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/data/instance:
uuid: "3be1115259f04b83a521dd37552145eb"
format_stamp: "Formatted at 2026-09-28 21:51:50 on dist-test-slave-jrck"
server_key: "7bea836d26e672b912f00e4d71d271b1"
server_key_iv: "94ba398941f73a084aea01661aeba323"
server_key_version: "encryptionkey@0"
I20260928 21:51:50.693532 16490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/wal/instance:
uuid: "3be1115259f04b83a521dd37552145eb"
format_stamp: "Formatted at 2026-09-28 21:51:50 on dist-test-slave-jrck"
server_key: "7bea836d26e672b912f00e4d71d271b1"
server_key_iv: "94ba398941f73a084aea01661aeba323"
server_key_version: "encryptionkey@0"
I20260928 21:51:50.694841 16490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:51:50.695643 16505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:50.695822 16490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:50.695946 16490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/wal
uuid: "3be1115259f04b83a521dd37552145eb"
format_stamp: "Formatted at 2026-09-28 21:51:50 on dist-test-slave-jrck"
server_key: "7bea836d26e672b912f00e4d71d271b1"
server_key_iv: "94ba398941f73a084aea01661aeba323"
server_key_version: "encryptionkey@0"
I20260928 21:51:50.696013 16490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:50.706467 16483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } attempt: 2
I20260928 21:51:50.706840 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.706729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:41289 (local address 127.0.0.1:54074)
0928 21:51:50.706778 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.706807 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 21:51:50.707177 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.707093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:41289 (local address 127.0.0.1:54090)
0928 21:51:50.707131 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.707156 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 21:51:50.707250 16483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.253:41289: Network error: Client connection negotiation failed: client connection to 127.3.0.253:41289: connect: Connection refused (error 111)
I20260928 21:51:50.715693 16490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:50.717519 16490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:50.717649 16490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:50.721462 16490 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.253:41289
I20260928 21:51:50.721508 16557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.253:41289 every 8 connection(s)
I20260928 21:51:50.721830 16490 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
I20260928 21:51:50.722671 16558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:50.724200 16558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:51:50.724493 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 16490
I20260928 21:51:50.724591 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-1/wal/instance
I20260928 21:51:50.724769 3075 external_mini_cluster.cc:1428] Setting key 51c0a9470ccc589338da24675bf85b9b
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.0.252@KRBTEST.COM; defaulting to no policy
I20260928 21:51:50.732761 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.724938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.3.0.254:39985)
0928 21:51:50.724979 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.724982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.725213 (+ 231us) server_negotiation.cc:407] Connection header received
0928 21:51:50.725687 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.725689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.725705 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.725763 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.726365 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.726863 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.727818 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.728026 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.728082 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.730094 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.730095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.730096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.730103 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.731455 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.731830 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.731831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.731831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.731886 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.732111 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.732112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.732112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.732227 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.732283 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.732578 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.732670 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:51:50.732918 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.724953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:39985 (local address 127.0.0.1:48292)
0928 21:51:50.725101 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.725137 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:50.725588 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:50.725831 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:50.725834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:50.726084 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:50.726275 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.726279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.726944 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.726946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.727735 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.727737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.728094 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.728095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.728211 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.728716 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:50.728725 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:50.730006 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:50.731542 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.731549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.731551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.731763 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.731940 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.731941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.731941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.732054 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.732281 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:50.732287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:50.732429 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 21:51:50.732547 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 21:51:50.732701 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.252@KRBTEST.COM" created.
I20260928 21:51:50.733469 16558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
I20260928 21:51:50.741425 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.733719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:41289 (local address 127.0.0.1:54106)
0928 21:51:50.733766 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.733775 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:50.733811 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:50.735040 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:50.735041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:50.735177 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:50.735225 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.735227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.736050 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.736051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.736619 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.736620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.737061 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.737061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.737120 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.737550 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:50.737554 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:50.738933 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:50.740522 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.740522 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.740523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.740667 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.740832 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.740832 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.740833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.740882 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.741133 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:50.741134 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:50.741169 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:51:50.741228 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:51:50.741324 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 21:51:50.741505 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.734751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.3.0.253:41289)
0928 21:51:50.734898 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.734903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.734922 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:51:50.734941 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.734943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.734954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.735031 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.735287 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.735981 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.736687 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.737001 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.737059 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.739009 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.739017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.739017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.739023 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.740424 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.740731 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.740734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.740735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.740781 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.740930 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.740931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.740931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.741075 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.741148 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.741257 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.741395 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260928 21:51:50.742077 16558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:51:50.742409 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.742289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55404)
0928 21:51:50.742332 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.742383 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 21:51:50.742549 16508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
I20260928 21:51:50.742923 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.742845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55406)
0928 21:51:50.742874 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.742901 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 21:51:50.743008 16558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:38837: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab.
Entry for principal kudu/127.3.0.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab.
I20260928 21:51:50.747133 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.252:38837
--webserver_interface=127.3.0.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab
--principal=kudu/127.3.0.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:39985,127.3.0.253:41289,127.3.0.252:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:51:50.792243 16558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } attempt: 1
I20260928 21:51:50.792778 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.792651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55410)
0928 21:51:50.792705 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.792747 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:51:50.793140 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.793014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55424)
0928 21:51:50.793080 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.793112 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 21:51:50.793242 16558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:38837: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
I20260928 21:51:50.798379 16483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } attempt: 3
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
I20260928 21:51:50.805500 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.798725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.3.0.253:41289)
0928 21:51:50.798779 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.798781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.798792 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:51:50.798855 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.798856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.798865 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.798912 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.799281 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.799785 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.800908 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.801132 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.801197 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.803127 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.803128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.803129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.803135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.804399 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.804801 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.804802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.804803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.804851 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.805033 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.805033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.805034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.805166 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.805215 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.805378 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.805435 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:51:50.805572 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.798674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:41289 (local address 127.0.0.1:54114)
0928 21:51:50.798729 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.798741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:50.798793 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:50.798923 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:50.798924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:50.799098 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:50.799189 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.799191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.799887 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.799888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.800801 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.800805 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.801205 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.801205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.801286 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.801721 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:50.801727 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:50.803048 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:50.804510 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.804511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.804512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.804733 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.804907 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.804908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.804908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.804975 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.805224 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:50.805225 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:50.805278 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 21:51:50.805350 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:51:50.805471 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:51:50.805989 16483 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:51:50.806308 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.806199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55430)
0928 21:51:50.806245 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.806281 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 21:51:50.806694 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.806607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55436)
0928 21:51:50.806649 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.806675 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 21:51:50.806783 16483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:38837: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
W20260928 21:51:50.829711 16565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:50.829923 16565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:50.829944 16565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:50.831064 16565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:51:50.831103 16565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:50.831116 16565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:50.831130 16565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:51:50.831151 16565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:51:50.832693 16565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:39985,127.3.0.253:41289,127.3.0.252:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab
--principal=kudu/127.3.0.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.252:38837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.0.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:50.832929 16565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:50.833137 16565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:50.837590 16570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:50.837595 16573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:50.837600 16571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:50.837924 16483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } attempt: 1
I20260928 21:51:50.838160 16565 server_base.cc:1034] running on GCE node
I20260928 21:51:50.838299 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.838196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55450)
0928 21:51:50.838240 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.838266 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 21:51:50.838406 16565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:50.838598 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.838492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55458)
0928 21:51:50.838533 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.838569 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 21:51:50.838682 16483 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:38837: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
I20260928 21:51:50.838733 16565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:50.839875 16565 hybrid_clock.cc:648] HybridClock initialized: now 1790632310839862 us; error 25 us; skew 500 ppm
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:50.842931 16565 init.cc:377] Logged in from keytab as kudu/127.3.0.252@KRBTEST.COM (short username kudu)
I20260928 21:51:50.843484 16565 webserver.cc:460] Webserver started at http://127.3.0.252:35549/ using document root <none> and password file <none>
I20260928 21:51:50.843747 16565 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:50.843813 16565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:50.843961 16565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:51:50.845395 16565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/data/instance:
uuid: "a46301cb46014702a389aab81b18145e"
format_stamp: "Formatted at 2026-09-28 21:51:50 on dist-test-slave-jrck"
server_key: "e2a73c75d98ca91eab04790d11214baa"
server_key_iv: "021c9bd9993c20d3e99984760b13e8de"
server_key_version: "encryptionkey@0"
I20260928 21:51:50.845791 16565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/wal/instance:
uuid: "a46301cb46014702a389aab81b18145e"
format_stamp: "Formatted at 2026-09-28 21:51:50 on dist-test-slave-jrck"
server_key: "e2a73c75d98ca91eab04790d11214baa"
server_key_iv: "021c9bd9993c20d3e99984760b13e8de"
server_key_version: "encryptionkey@0"
I20260928 21:51:50.847453 16565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:51:50.848251 16580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:50.848418 16565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:50.848529 16565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/wal
uuid: "a46301cb46014702a389aab81b18145e"
format_stamp: "Formatted at 2026-09-28 21:51:50 on dist-test-slave-jrck"
server_key: "e2a73c75d98ca91eab04790d11214baa"
server_key_iv: "021c9bd9993c20d3e99984760b13e8de"
server_key_version: "encryptionkey@0"
I20260928 21:51:50.848595 16565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:50.861372 16558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } attempt: 2
I20260928 21:51:50.861795 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.861684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55466)
0928 21:51:50.861731 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.861768 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 21:51:50.862062 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.861997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55470)
0928 21:51:50.862022 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.862044 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260928 21:51:50.862175 16558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:38837: Network error: Client connection negotiation failed: client connection to 127.3.0.252:38837: connect: Connection refused (error 111)
I20260928 21:51:50.863516 16565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:50.865286 16565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:50.865401 16565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:50.869227 16565 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.252:38837
I20260928 21:51:50.869267 16632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.252:38837 every 8 connection(s)
I20260928 21:51:50.869594 16565 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
I20260928 21:51:50.870428 16633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:50.871563 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 16565
I20260928 21:51:50.871651 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790632139349757-3075-0/minicluster-data/master-2/wal/instance
I20260928 21:51:50.871827 3075 external_mini_cluster.cc:1428] Setting key c88d165ff3a68334812e53273b0b6180
I20260928 21:51:50.872052 16633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
I20260928 21:51:50.880816 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.872725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.3.0.254:39985)
0928 21:51:50.872763 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.872765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.872935 (+ 170us) server_negotiation.cc:407] Connection header received
0928 21:51:50.873190 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.873192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.873203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.873254 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.873933 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.874419 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.875358 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.875567 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.875616 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.877896 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.877897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.877898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.877904 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.879283 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.879775 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.879776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.879777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.879836 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.880106 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.880107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.880108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.880230 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.880305 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.880644 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.880724 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:51:50.881001 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.872665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:39985 (local address 127.0.0.1:48300)
0928 21:51:50.872817 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.872849 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:50.873101 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:50.873276 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:50.873279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:50.873627 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:50.873864 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.873870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.874486 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.874487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.875272 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.875274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.875637 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.875638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.875791 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.876308 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:50.876316 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:50.877819 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:50.879353 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.879361 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.879364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.879707 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.879894 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.879895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.879895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.880015 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.880321 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:50.880328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:50.880498 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 21:51:50.880610 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 21:51:50.880764 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260928 21:51:50.881325 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.872304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.3.0.252:38837)
0928 21:51:50.872503 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.872513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.872549 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:51:50.873095 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.873102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.873124 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.873219 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.874500 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.875265 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.876145 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.876555 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.876609 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.878717 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.878725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.878725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.878731 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.880230 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.880554 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.880557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.880558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.880609 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.880769 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.880770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.880770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.880918 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.881057 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.881146 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.881221 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 21:51:50.881582 16633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:50.887603 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.881885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.3.0.253:41289)
0928 21:51:50.881926 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.881928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.881937 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:51:50.881978 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.881978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.881989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.882039 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.882276 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.882731 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.883379 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.883574 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.883630 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.885445 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.885446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.885446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.885453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.886570 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.886913 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.886915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.886915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.886975 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.887150 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.887151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.887151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.887251 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.887297 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.887470 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.887542 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:51:50.887598 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.881820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:41289 (local address 127.0.0.1:54116)
0928 21:51:50.881854 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.881864 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:50.881904 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:50.882042 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:50.882043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:50.882168 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:50.882206 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.882207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.882793 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.882794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.883303 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.883304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.883633 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.883633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.883693 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.884035 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:50.884039 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:50.885370 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:50.886625 (+ 1255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.886626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.886626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.886817 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.887035 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.887035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.887036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.887093 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.887305 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:50.887306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:50.887345 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 21:51:50.887416 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:51:50.887499 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 21:51:50.888006 16633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:51:50.888797 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.881980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.3.0.254:39985)
0928 21:51:50.882022 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.882025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.882032 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:51:50.882081 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.882082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.882091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.882139 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.882562 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.882993 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.883612 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.883858 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.883919 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.886224 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.886226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.886226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.886232 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.887691 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.888011 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.888013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.888013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.888065 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.888247 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.888248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.888248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.888383 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.888609 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.888645 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.888711 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
I20260928 21:51:50.894013 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.888243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55492)
0928 21:51:50.888278 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.888288 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:50.888320 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:50.888420 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:50.888421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:50.888570 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:50.888594 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.888596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.889364 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.889365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.889833 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.889834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.890154 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.890155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.890198 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.890555 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:50.890560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:50.891781 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:50.893176 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.893177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.893177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.893319 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.893466 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.893466 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.893467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.893532 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.893727 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:50.893728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:50.893762 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 21:51:50.893822 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:51:50.893912 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 21:51:50.894061 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.888287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.3.0.252:38837)
0928 21:51:50.888317 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.888319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.888325 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:51:50.888370 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.888371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.888378 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.888420 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.888654 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.889310 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.889898 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.890102 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.890151 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.891849 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.891850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.891850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.891856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.893109 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.893374 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.893375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.893375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.893422 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.893581 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.893581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.893582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.893677 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.893728 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.893845 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.893957 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":16}
I20260928 21:51:50.895290 16633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e: Bootstrap starting.
I20260928 21:51:50.896108 16633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:50.896117 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.889319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.3.0.253:41289)
0928 21:51:50.889350 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.889352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.889396 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:51:50.889467 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.889468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.889476 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.889518 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.889906 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.890363 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.891211 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.891518 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.891588 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.893706 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.893707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.893707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.893713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.895156 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.895454 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.895455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.895456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.895506 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.895663 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.895664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.895664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.895764 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.895933 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.895968 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.896035 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:51:50.896409 16633 log.cc:824] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:50.897189 16633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e: No bootstrap required, opened a new log
I20260928 21:51:50.897743 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:51:50.899068 16633 raft_consensus.cc:348] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } }
I20260928 21:51:50.899232 16633 raft_consensus.cc:374] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:50.899271 16633 raft_consensus.cc:729] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46301cb46014702a389aab81b18145e, State: Initialized, Role: FOLLOWER
I20260928 21:51:50.899391 16633 consensus_queue.cc:260] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } }
I20260928 21:51:50.899688 16633 sys_catalog.cc:564] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:51:50.899652 16646 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } } }
I20260928 21:51:50.899763 16646 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:51:50.901476 16657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:51:50.901654 16657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:51:50.902172 16565 master_runner.cc:186] Error getting master registration for 127.3.0.254:39985: OK, instance_id { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" instance_seqno: 1790632310572582 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:51:50.904366 16565 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:51:50.906987 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.899346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55496 (local address 127.3.0.252:38837)
0928 21:51:50.899378 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.899380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.899946 (+ 566us) server_negotiation.cc:407] Connection header received
0928 21:51:50.900003 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.900004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.900014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.900055 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.900681 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.901394 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.902693 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.902940 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.903000 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.904194 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.904195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.904195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.904201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.905674 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.906020 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.906021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.906021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.906076 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.906263 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.906264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.906264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.906407 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.906493 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.906845 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.906910 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:51:50.907752 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.899367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.3.0.254:39985)
0928 21:51:50.899517 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.899519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.899526 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:51:50.899544 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.899545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.899553 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.899606 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.900370 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.901095 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.902167 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.902430 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.902471 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.905171 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.905172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.905173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.905180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.906649 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.906980 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.906981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.906981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.907034 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.907230 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.907231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.907231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.907351 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.907415 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.907594 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.907660 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260928 21:51:50.909022 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.899174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.3.0.253:41289)
0928 21:51:50.899209 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.899210 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.900075 (+ 865us) server_negotiation.cc:407] Connection header received
0928 21:51:50.900157 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.900158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.900167 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.900194 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.900772 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.901447 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.902697 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.903426 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.903469 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.905657 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.905658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.905658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.905665 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.907332 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.908078 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.908079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.908080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.908132 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.908320 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.908321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.908321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.908406 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.908866 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.908899 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.908955 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:51:50.913818 16483 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } attempt: 2
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
I20260928 21:51:50.920217 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.914139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.3.0.252:38837)
0928 21:51:50.914180 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.914182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.914191 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:51:50.914212 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.914212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.914219 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.914261 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.914519 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.914946 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.915856 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.916032 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.916080 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.918114 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.918115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.918115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.918121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.919294 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.919636 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.919636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.919637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.919678 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.919846 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.919847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.919847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.919932 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.919975 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.920107 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.920159 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:51:50.920274 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.914066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55500)
0928 21:51:50.914117 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.914129 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:50.914172 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:50.914265 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:50.914266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:50.914418 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:50.914461 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.914463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.915003 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.915004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.915788 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.915789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.916095 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.916096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.916153 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.916771 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:50.916775 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:50.918040 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:50.919369 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.919370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.919370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.919572 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.919725 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.919726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.919726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.919786 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.919978 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:50.919979 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:50.920023 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 21:51:50.920091 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:51:50.920176 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:51:50.921868 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b: Bootstrap starting.
I20260928 21:51:50.922585 16483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:50.922871 16483 log.cc:824] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:50.923802 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b: No bootstrap required, opened a new log
I20260928 21:51:50.925230 16483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } }
I20260928 21:51:50.925336 16483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:50.925367 16483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556cbc69eea4482ca523179af8f0275b, State: Initialized, Role: FOLLOWER
I20260928 21:51:50.925482 16483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } }
I20260928 21:51:50.925766 16483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:51:50.925745 16660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } } }
I20260928 21:51:50.925827 16660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:51:50.927209 16671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:51:50.927259 16671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:51:50.927918 16415 master_runner.cc:186] Error getting master registration for 127.3.0.253:41289: OK, instance_id { permanent_uuid: "3be1115259f04b83a521dd37552145eb" instance_seqno: 1790632310720360 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:51:50.930181 16415 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:51:50.953329 16558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } attempt: 3
Sep 28 21:51:50 dist-test-slave-jrck krb5kdc[16402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632310, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
I20260928 21:51:50.961167 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.953662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:38837 (local address 127.0.0.1:55516)
0928 21:51:50.953718 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:50.953730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:50.953770 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:50.953918 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:50.953919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:50.954105 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:50.954157 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.954159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.955051 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.955052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.955870 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:50.955873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.956319 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.956320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:50.956380 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.956874 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:50.956879 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:50.958353 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:50.960125 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.960126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.960126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.960340 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.960570 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:50.960570 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:50.960571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:50.960626 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:50.960869 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:50.960870 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:50.960913 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:51:50.960977 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:51:50.961072 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:51:50.961511 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:50.953708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.3.0.252:38837)
0928 21:51:50.953752 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:50.953754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:50.953763 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:51:50.953832 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:50.953833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:50.953843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:50.953909 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:50.954223 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.954972 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.955951 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:50.956248 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:50.956317 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:50.958443 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:50.958445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:50.958445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:50.958453 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:50.960043 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.960424 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.960425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.960425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.960507 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:50.960679 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:50.960680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:50.960680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:50.960799 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:50.960849 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:50.961354 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 21:51:50.961430 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:51:50.963028 16558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb: Bootstrap starting.
I20260928 21:51:50.963788 16558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:50.964087 16558 log.cc:824] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:50.964999 16558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb: No bootstrap required, opened a new log
I20260928 21:51:50.966969 16558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } }
I20260928 21:51:50.967113 16558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:50.967144 16558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be1115259f04b83a521dd37552145eb, State: Initialized, Role: FOLLOWER
I20260928 21:51:50.967283 16558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } }
I20260928 21:51:50.967621 16558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:51:50.967583 16674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } } }
I20260928 21:51:50.967782 16674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:51:50.968238 16511 tablet.cc:2369] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:51:50.969352 16685 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:51:50.969472 16685 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:51:50.971601 16490 master_runner.cc:263] No leader master found from master 3be1115259f04b83a521dd37552145eb
I20260928 21:51:50.971652 16490 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 21:51:51.070662 16436 tablet.cc:2369] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:51:51.115993 16586 tablet.cc:2369] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:51:51.155205 16660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:51:51.155328 16660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } }
I20260928 21:51:51.155599 16660 leader_election.cc:288] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3be1115259f04b83a521dd37552145eb (127.3.0.253:41289), a46301cb46014702a389aab81b18145e (127.3.0.252:38837)
I20260928 21:51:51.155896 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "556cbc69eea4482ca523179af8f0275b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46301cb46014702a389aab81b18145e" is_pre_election: true
I20260928 21:51:51.155910 16532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "556cbc69eea4482ca523179af8f0275b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be1115259f04b83a521dd37552145eb" is_pre_election: true
I20260928 21:51:51.156047 16607 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556cbc69eea4482ca523179af8f0275b in term 0.
I20260928 21:51:51.156049 16532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556cbc69eea4482ca523179af8f0275b in term 0.
I20260928 21:51:51.156229 16433 leader_election.cc:302] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556cbc69eea4482ca523179af8f0275b, a46301cb46014702a389aab81b18145e; no voters:
I20260928 21:51:51.156355 16660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:51:51.156411 16660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:51:51.156427 16660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:51.157130 16660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } }
I20260928 21:51:51.157236 16660 leader_election.cc:288] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [CANDIDATE]: Term 1 election: Requested vote from peers 3be1115259f04b83a521dd37552145eb (127.3.0.253:41289), a46301cb46014702a389aab81b18145e (127.3.0.252:38837)
I20260928 21:51:51.157423 16607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "556cbc69eea4482ca523179af8f0275b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46301cb46014702a389aab81b18145e"
I20260928 21:51:51.157423 16532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "556cbc69eea4482ca523179af8f0275b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be1115259f04b83a521dd37552145eb"
I20260928 21:51:51.157495 16607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:51.157500 16532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:51.158170 16607 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556cbc69eea4482ca523179af8f0275b in term 1.
I20260928 21:51:51.158195 16532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556cbc69eea4482ca523179af8f0275b in term 1.
I20260928 21:51:51.158337 16433 leader_election.cc:302] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556cbc69eea4482ca523179af8f0275b, a46301cb46014702a389aab81b18145e; no voters:
I20260928 21:51:51.158464 16660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:51:51.158587 16660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [term 1 LEADER]: Becoming Leader. State: Replica: 556cbc69eea4482ca523179af8f0275b, State: Running, Role: LEADER
I20260928 21:51:51.158676 16660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } }
I20260928 21:51:51.159042 16687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556cbc69eea4482ca523179af8f0275b. Latest consensus state: current_term: 1 leader_uuid: "556cbc69eea4482ca523179af8f0275b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } } }
I20260928 21:51:51.159116 16687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:51.159304 16689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:51:51.159474 16689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:51:51.160728 16607 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [term 1 FOLLOWER]: Refusing update from remote peer 556cbc69eea4482ca523179af8f0275b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:51:51.160758 16532 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [term 1 FOLLOWER]: Refusing update from remote peer 556cbc69eea4482ca523179af8f0275b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:51:51.161032 16687 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:51:51.161146 16660 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:51:51.162204 16646 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556cbc69eea4482ca523179af8f0275b. Latest consensus state: current_term: 1 leader_uuid: "556cbc69eea4482ca523179af8f0275b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } } }
I20260928 21:51:51.162295 16646 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:51:51.162285 16674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556cbc69eea4482ca523179af8f0275b. Latest consensus state: current_term: 1 leader_uuid: "556cbc69eea4482ca523179af8f0275b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } } }
I20260928 21:51:51.162369 16674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:51:51.163094 16660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "556cbc69eea4482ca523179af8f0275b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } } }
I20260928 21:51:51.163103 16687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "556cbc69eea4482ca523179af8f0275b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } } }
I20260928 21:51:51.163218 16687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:51.163183 16660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:51.163278 16646 sys_catalog.cc:455] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "556cbc69eea4482ca523179af8f0275b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } } }
I20260928 21:51:51.163293 16674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "556cbc69eea4482ca523179af8f0275b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556cbc69eea4482ca523179af8f0275b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39985 } } peers { permanent_uuid: "3be1115259f04b83a521dd37552145eb" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 41289 } } peers { permanent_uuid: "a46301cb46014702a389aab81b18145e" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 38837 } } }
I20260928 21:51:51.163331 16691 mvcc.cc:205] Tried to move back new op lower bound from 7334429946512375808 to 7334429946506502144. Current Snapshot: MvccSnapshot[applied={T|T < 7334429946512375808}]
I20260928 21:51:51.163353 16674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:51:51.163353 16646 sys_catalog.cc:458] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:51:51.163499 16692 mvcc.cc:205] Tried to move back new op lower bound from 7334429946512375808 to 7334429946506502144. Current Snapshot: MvccSnapshot[applied={T|T < 7334429946512375808}]
I20260928 21:51:51.163748 16689 catalog_manager.cc:1347] Generated new cluster ID: a28910a5780c406b837b0a82c334cbcf
I20260928 21:51:51.163792 16689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:51:51.169093 16689 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:51:51.169646 16689 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:51:51.177605 16689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 556cbc69eea4482ca523179af8f0275b: Generated new TSK 0
I20260928 21:51:51.177834 16689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:51:51.902107 16657 catalog_manager.cc:1259] Loaded cluster ID: a28910a5780c406b837b0a82c334cbcf
I20260928 21:51:51.902171 16657 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e: loading cluster ID for follower catalog manager: success
I20260928 21:51:51.903306 16657 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e: acquiring CA information for follower catalog manager: success
I20260928 21:51:51.903664 16657 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a46301cb46014702a389aab81b18145e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:51:51.969914 16685 catalog_manager.cc:1259] Loaded cluster ID: a28910a5780c406b837b0a82c334cbcf
I20260928 21:51:51.969990 16685 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb: loading cluster ID for follower catalog manager: success
I20260928 21:51:51.970824 16685 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb: acquiring CA information for follower catalog manager: success
I20260928 21:51:51.971135 16685 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3be1115259f04b83a521dd37552145eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:51:52.237228 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 16415
I20260928 21:51:52.241815 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 16490
I20260928 21:51:52.245980 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 16565
2026-09-28T21:51:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2845 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:51:52 dist-test-slave-jrck krb5kdc[16704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:51:52 dist-test-slave-jrck krb5kdc[16704](info): set up 2 sockets
Sep 28 21:51:52 dist-test-slave-jrck krb5kdc[16704](info): commencing operation
krb5kdc: starting...
W20260928 21:51:53.188231 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:51:53 dist-test-slave-jrck krb5kdc[16704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:51:53.295818 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33961
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:42659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:33961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:53.374599 16717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:53.374835 16717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:53.374866 16717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:53.376044 16717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:51:53.376087 16717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:53.376106 16717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:53.376122 16717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:51:53.376137 16717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:51:53.377744 16717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:33961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:53.378002 16717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:53.378223 16717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:53.382913 16723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:53.382941 16717 server_base.cc:1034] running on GCE node
W20260928 21:51:53.382913 16725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:53.382943 16722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:53.383332 16717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:53.383669 16717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:53.384821 16717 hybrid_clock.cc:648] HybridClock initialized: now 1790632313384802 us; error 35 us; skew 500 ppm
Sep 28 21:51:53 dist-test-slave-jrck krb5kdc[16704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:53.387220 16717 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:51:53.387676 16717 webserver.cc:460] Webserver started at http://127.3.0.254:34529/ using document root <none> and password file <none>
I20260928 21:51:53.387878 16717 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:53.387925 16717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:53.388051 16717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:51:53.389132 16717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "2a8dc65de579411fa5ecdca063703c68"
format_stamp: "Formatted at 2026-09-28 21:51:53 on dist-test-slave-jrck"
server_key: "d28051cd02fe431f016c1a8f763207d3"
server_key_iv: "9b04092aafab6ea694cc33653618d072"
server_key_version: "encryptionkey@0"
I20260928 21:51:53.389467 16717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "2a8dc65de579411fa5ecdca063703c68"
format_stamp: "Formatted at 2026-09-28 21:51:53 on dist-test-slave-jrck"
server_key: "d28051cd02fe431f016c1a8f763207d3"
server_key_iv: "9b04092aafab6ea694cc33653618d072"
server_key_version: "encryptionkey@0"
I20260928 21:51:53.390753 16717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:51:53.391466 16732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:53.391637 16717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:53.391723 16717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "2a8dc65de579411fa5ecdca063703c68"
format_stamp: "Formatted at 2026-09-28 21:51:53 on dist-test-slave-jrck"
server_key: "d28051cd02fe431f016c1a8f763207d3"
server_key_iv: "9b04092aafab6ea694cc33653618d072"
server_key_version: "encryptionkey@0"
I20260928 21:51:53.391781 16717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:53.400799 16717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:53.402554 16717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:53.402697 16717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:53.406378 16717 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:33961
I20260928 21:51:53.406407 16784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:33961 every 8 connection(s)
I20260928 21:51:53.406733 16717 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:51:53.407532 16785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:53.409783 16785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68: Bootstrap starting.
I20260928 21:51:53.409993 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 16717
I20260928 21:51:53.410080 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:51:53.410248 3075 external_mini_cluster.cc:1428] Setting key f8aa7be728d469352b4630a55c182df9
I20260928 21:51:53.410413 16785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:53.410699 16785 log.cc:824] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:53.411392 16785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68: No bootstrap required, opened a new log
I20260928 21:51:53.413662 16785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8dc65de579411fa5ecdca063703c68" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33961 } }
I20260928 21:51:53.413836 16785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:53.413863 16785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8dc65de579411fa5ecdca063703c68, State: Initialized, Role: FOLLOWER
I20260928 21:51:53.413995 16785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8dc65de579411fa5ecdca063703c68" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33961 } }
I20260928 21:51:53.414067 16785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:51:53.414088 16785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:51:53.414139 16785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:53.414733 16785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8dc65de579411fa5ecdca063703c68" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33961 } }
I20260928 21:51:53.414842 16785 leader_election.cc:302] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a8dc65de579411fa5ecdca063703c68; no voters:
I20260928 21:51:53.415000 16785 leader_election.cc:288] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:51:53.415110 16790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:51:53.415266 16785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:51:53.415272 16790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [term 1 LEADER]: Becoming Leader. State: Replica: 2a8dc65de579411fa5ecdca063703c68, State: Running, Role: LEADER
I20260928 21:51:53.415388 16790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8dc65de579411fa5ecdca063703c68" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33961 } }
I20260928 21:51:53.416615 16791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a8dc65de579411fa5ecdca063703c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8dc65de579411fa5ecdca063703c68" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33961 } } }
I20260928 21:51:53.416719 16791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:53.417150 16792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a8dc65de579411fa5ecdca063703c68. Latest consensus state: current_term: 1 leader_uuid: "2a8dc65de579411fa5ecdca063703c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a8dc65de579411fa5ecdca063703c68" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33961 } } }
Sep 28 21:51:53 dist-test-slave-jrck krb5kdc[16704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:53.417238 16792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:53.417160 16799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:51:53.417960 16799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:51:53.419541 16799 catalog_manager.cc:1347] Generated new cluster ID: 3a2dbab5adad4f349da9755006542b90
I20260928 21:51:53.419588 16799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:51:53.421543 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:53.410710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.3.0.254:33961)
0928 21:51:53.410905 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:53.410914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:53.410945 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:51:53.411468 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:53.411478 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:53.411548 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:53.411709 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:53.412264 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:53.413666 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:53.414529 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:53.415149 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:53.415295 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:53.417962 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:53.417966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:53.417966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:53.417978 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:53.419768 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:53.420121 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:53.420124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:53.420125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:53.420221 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:53.420384 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:53.420385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:53.420385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:53.420686 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:53.420818 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:53.421214 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 21:51:53.421340 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
I20260928 21:51:53.422492 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:51:53.428587 16799 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:51:53.429276 16799 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:51:53.436976 16799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a8dc65de579411fa5ecdca063703c68: Generated new TSK 0
I20260928 21:51:53.437139 16799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:51:53.476183 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 16717
2026-09-28T21:51:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1229 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:51:53 dist-test-slave-jrck krb5kdc[16812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:51:53 dist-test-slave-jrck krb5kdc[16812](info): set up 2 sockets
Sep 28 21:51:53 dist-test-slave-jrck krb5kdc[16812](info): commencing operation
krb5kdc: starting...
W20260928 21:51:54.416363 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:51:54 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:51:54.522267 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:36185
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:54.601061 16825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:54.601248 16825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:54.601269 16825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:54.602392 16825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:51:54.602432 16825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:54.602444 16825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:54.602466 16825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:51:54.602483 16825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:51:54.603952 16825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:54.604171 16825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:54.604372 16825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:54.608736 16830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:54.608757 16833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:54.608742 16831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:54.608855 16825 server_base.cc:1034] running on GCE node
I20260928 21:51:54.609179 16825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:54.609438 16825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:54.610565 16825 hybrid_clock.cc:648] HybridClock initialized: now 1790632314610557 us; error 26 us; skew 500 ppm
Sep 28 21:51:54 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:54.612835 16825 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:51:54.613207 16825 webserver.cc:460] Webserver started at http://127.3.0.254:37581/ using document root <none> and password file <none>
I20260928 21:51:54.613387 16825 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:54.613423 16825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:54.613512 16825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:51:54.614485 16825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "40d07be0c92a49b58eceb8bb2e31d6a5"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "4bdd7a90df6a3421d9bffaa6882c2973"
server_key_iv: "0cd5837a15ff0dad0ed3e2816798dadc"
server_key_version: "encryptionkey@0"
I20260928 21:51:54.614835 16825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "40d07be0c92a49b58eceb8bb2e31d6a5"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "4bdd7a90df6a3421d9bffaa6882c2973"
server_key_iv: "0cd5837a15ff0dad0ed3e2816798dadc"
server_key_version: "encryptionkey@0"
I20260928 21:51:54.616053 16825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:51:54.616714 16840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:54.616851 16825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:51:54.616948 16825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "40d07be0c92a49b58eceb8bb2e31d6a5"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "4bdd7a90df6a3421d9bffaa6882c2973"
server_key_iv: "0cd5837a15ff0dad0ed3e2816798dadc"
server_key_version: "encryptionkey@0"
I20260928 21:51:54.617002 16825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:54.625494 16825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:54.627456 16825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:54.627561 16825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:54.631081 16825 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:36185
I20260928 21:51:54.631124 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:36185 every 8 connection(s)
I20260928 21:51:54.631426 16825 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:51:54.632362 16893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:54.634575 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Bootstrap starting.
I20260928 21:51:54.635082 16893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:54.635304 16893 log.cc:824] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:54.635911 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: No bootstrap required, opened a new log
I20260928 21:51:54.636350 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 16825
I20260928 21:51:54.636441 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:51:54.636629 3075 external_mini_cluster.cc:1428] Setting key 61f750baf5401e0bf395d08ca2060359
I20260928 21:51:54.637928 16893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:54.638099 16893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:54.638134 16893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d07be0c92a49b58eceb8bb2e31d6a5, State: Initialized, Role: FOLLOWER
I20260928 21:51:54.638255 16893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:54.638326 16893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:51:54.638347 16893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:51:54.638398 16893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:54.639396 16893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:54.639500 16893 leader_election.cc:302] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40d07be0c92a49b58eceb8bb2e31d6a5; no voters:
I20260928 21:51:54.639638 16893 leader_election.cc:288] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:51:54.639675 16898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:51:54.639814 16898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 1 LEADER]: Becoming Leader. State: Replica: 40d07be0c92a49b58eceb8bb2e31d6a5, State: Running, Role: LEADER
I20260928 21:51:54.639866 16893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:51:54.639904 16898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:54.640195 16898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d07be0c92a49b58eceb8bb2e31d6a5. Latest consensus state: current_term: 1 leader_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } } }
I20260928 21:51:54.640264 16898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:54.640424 16900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } } }
I20260928 21:51:54.640506 16900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:54.640609 16904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:51:54.641258 16904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:51:54.642730 16904 catalog_manager.cc:1347] Generated new cluster ID: b8d3554a21504e99bf0fc91df0fc6b05
I20260928 21:51:54.642776 16904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:51:54 dist-test-slave-jrck krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:54.648010 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:54.637073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.3.0.254:36185)
0928 21:51:54.637256 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:54.637266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:54.637300 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:51:54.637891 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:54.637900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:54.637961 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:54.638108 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:54.638788 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:54.640153 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:54.641258 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:54.641922 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:54.642025 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:54.644641 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:54.644644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:54.644645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:54.644655 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:54.646473 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:54.646856 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:54.646859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:54.646859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:54.646979 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:54.647177 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:54.647177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:54.647178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:54.647455 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:54.647590 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:54.647697 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:51:54.647821 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260928 21:51:54.656358 16904 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:51:54.657073 16904 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:51:54.668083 16904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Generated new TSK 0
I20260928 21:51:54.668264 16904 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:51:54.689010 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36185
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:54.767196 16919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:54.767396 16919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:54.767426 16919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:54.768563 16919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:54.768599 16919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:54.768633 16919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:51:54.770017 16919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:54.770231 16919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:54.770437 16919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:54.774955 16925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:54.774966 16924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:54.775002 16919 server_base.cc:1034] running on GCE node
W20260928 21:51:54.774989 16927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:54.775314 16919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:54.775528 16919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:54.776655 16919 hybrid_clock.cc:648] HybridClock initialized: now 1790632314776638 us; error 32 us; skew 500 ppm
Sep 28 21:51:54 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:54.779011 16919 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:51:54.779451 16919 webserver.cc:460] Webserver started at http://127.3.0.193:39613/ using document root <none> and password file <none>
I20260928 21:51:54.779654 16919 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:54.779699 16919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:54.779821 16919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:51:54.780896 16919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "20b3ae98089b444aaddde508a09cc75b"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "daf3149c7407a562b74937ccffd4df44"
server_key_iv: "666a3d800480231728c63cbc8c72d617"
server_key_version: "encryptionkey@0"
I20260928 21:51:54.781208 16919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "20b3ae98089b444aaddde508a09cc75b"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "daf3149c7407a562b74937ccffd4df44"
server_key_iv: "666a3d800480231728c63cbc8c72d617"
server_key_version: "encryptionkey@0"
I20260928 21:51:54.782425 16919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:51:54.783110 16934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:54.783255 16919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:54.783325 16919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "20b3ae98089b444aaddde508a09cc75b"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "daf3149c7407a562b74937ccffd4df44"
server_key_iv: "666a3d800480231728c63cbc8c72d617"
server_key_version: "encryptionkey@0"
I20260928 21:51:54.783391 16919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:54.796550 16919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:54.798143 16919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:54.798267 16919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:54.798470 16919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:54.798748 16919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:54.798780 16919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:54.798807 16919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:54.798831 16919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:54.804301 16919 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:34351
I20260928 21:51:54.804358 17047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:34351 every 8 connection(s)
I20260928 21:51:54.804667 16919 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:51:54 dist-test-slave-jrck krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:54.812670 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:54.805316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:47865 (local address 127.3.0.254:36185)
0928 21:51:54.805367 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:54.805369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:54.805600 (+ 231us) server_negotiation.cc:407] Connection header received
0928 21:51:54.806064 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:54.806065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:54.806074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:54.806122 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:54.806978 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:54.807475 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:54.808371 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:54.808612 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:54.808656 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:54.810518 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:54.810519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:54.810520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:54.810526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:54.811595 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:54.811921 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:54.811921 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:54.811922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:54.811968 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:54.812136 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:54.812137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:54.812137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:54.812228 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:54.812269 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:54.812530 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 21:51:54.812605 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:51:54.812783 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:54.805289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:47865)
0928 21:51:54.805466 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:54.805512 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:54.805990 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:54.806179 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:54.806183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:54.806726 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:54.806908 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:54.806913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:54.807539 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:54.807539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:54.808308 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:54.808311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:54.808673 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:54.808674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:54.808771 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:54.809157 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:54.809165 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:54.810435 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:54.811657 (+ 1222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:54.811662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:54.811664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:54.811865 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:54.812013 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:54.812013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:54.812014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:54.812097 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:54.812268 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:54.812270 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:54.812373 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 21:51:54.812474 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:51:54.812609 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260928 21:51:54.813249 17048 heartbeater.cc:347] Connected to a master server at 127.3.0.254:36185
I20260928 21:51:54.813334 17048 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:54.813519 17048 heartbeater.cc:511] Master 127.3.0.254:36185 requested a full tablet report, sending...
I20260928 21:51:54.813541 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 16919
I20260928 21:51:54.813634 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:51:54.813822 3075 external_mini_cluster.cc:1428] Setting key f0d93eb65e2d8f489d631de6d5fef56e
I20260928 21:51:54.814011 16857 ts_manager.cc:194] Registered new tserver with Master: 20b3ae98089b444aaddde508a09cc75b (127.3.0.193:34351)
I20260928 21:51:54.814728 16857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:47865
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:51:54.834673 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36185
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:54.912961 17053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:54.913143 17053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:54.913163 17053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:54.914250 17053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:54.914288 17053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:54.914314 17053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:51:54.915736 17053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:54.915946 17053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:54.916149 17053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:54.920763 17058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:54.920733 17059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:54.920753 17061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:54.921092 17053 server_base.cc:1034] running on GCE node
I20260928 21:51:54.921259 17053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:54.921485 17053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:54.922627 17053 hybrid_clock.cc:648] HybridClock initialized: now 1790632314922614 us; error 32 us; skew 500 ppm
Sep 28 21:51:54 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:54.924896 17053 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:51:54.925256 17053 webserver.cc:460] Webserver started at http://127.3.0.194:43353/ using document root <none> and password file <none>
I20260928 21:51:54.925422 17053 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:54.925460 17053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:54.925547 17053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:51:54.926494 17053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "89225913bb0147d59cfe79b17ade11d0"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "ef212f4e9f2b9802d9585a22b93903f9"
server_key_iv: "29b022a9e3adeab40705f30dffc79a1b"
server_key_version: "encryptionkey@0"
I20260928 21:51:54.926766 17053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "89225913bb0147d59cfe79b17ade11d0"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "ef212f4e9f2b9802d9585a22b93903f9"
server_key_iv: "29b022a9e3adeab40705f30dffc79a1b"
server_key_version: "encryptionkey@0"
I20260928 21:51:54.927866 17053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:51:54.928516 17068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:54.928658 17053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:54.928733 17053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "89225913bb0147d59cfe79b17ade11d0"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "ef212f4e9f2b9802d9585a22b93903f9"
server_key_iv: "29b022a9e3adeab40705f30dffc79a1b"
server_key_version: "encryptionkey@0"
I20260928 21:51:54.928798 17053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:54.952508 17053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:54.954221 17053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:54.954336 17053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:54.954557 17053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:54.954885 17053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:54.954922 17053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:54.954943 17053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:54.954967 17053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:54.960283 17053 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:43029
I20260928 21:51:54.960340 17181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:43029 every 8 connection(s)
I20260928 21:51:54.960645 17053 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:51:54 dist-test-slave-jrck krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:54.968783 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:54.961289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:46005 (local address 127.3.0.254:36185)
0928 21:51:54.961350 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:54.961352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:54.961564 (+ 212us) server_negotiation.cc:407] Connection header received
0928 21:51:54.962020 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:54.962021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:54.962033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:54.962083 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:54.962948 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:54.963453 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:54.964356 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:54.964550 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:54.964590 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:54.966471 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:54.966472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:54.966473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:54.966479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:54.967590 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:54.967940 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:54.967941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:54.967942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:54.967994 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:54.968183 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:54.968183 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:54.968184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:54.968287 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:54.968338 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:54.968651 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 21:51:54.968718 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:51:54.968926 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 17053
I20260928 21:51:54.969002 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:54.961269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:46005)
0928 21:51:54.961451 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:54.961487 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:54.961947 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:54.962136 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:54.962139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:54.962692 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:54.962882 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:54.962888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:54.963520 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:54.963520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:54.964291 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:54.964292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:54.964603 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:54.964604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:54.964696 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:54.965094 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:54.965101 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:54.966388 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:54.967655 (+ 1267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:54.967661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:54.967663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:54.967874 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:54.968046 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:54.968046 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:54.968047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:54.968139 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:54.968375 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:54.968381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:54.968540 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 21:51:54.968635 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:51:54.968777 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260928 21:51:54.969033 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:51:54.969231 3075 external_mini_cluster.cc:1428] Setting key c50b0564b501b228f3727008931329d3
I20260928 21:51:54.969435 17182 heartbeater.cc:347] Connected to a master server at 127.3.0.254:36185
I20260928 21:51:54.969512 17182 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:54.969663 17182 heartbeater.cc:511] Master 127.3.0.254:36185 requested a full tablet report, sending...
I20260928 21:51:54.970005 16857 ts_manager.cc:194] Registered new tserver with Master: 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029)
I20260928 21:51:54.970358 16857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:46005
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:51:54.990492 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36185
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:55.070118 17187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:55.070327 17187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:55.070349 17187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:55.071467 17187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:55.071506 17187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:55.071532 17187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:51:55.072930 17187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:55.073134 17187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:55.073362 17187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:55.077827 17192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:55.077803 17193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:55.077814 17195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:55.078118 17187 server_base.cc:1034] running on GCE node
I20260928 21:51:55.078354 17187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:55.078574 17187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:55.079710 17187 hybrid_clock.cc:648] HybridClock initialized: now 1790632315079698 us; error 28 us; skew 500 ppm
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:55.082510 17187 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:51:55.082929 17187 webserver.cc:460] Webserver started at http://127.3.0.195:42419/ using document root <none> and password file <none>
I20260928 21:51:55.083127 17187 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:55.083170 17187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:55.083300 17187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:51:55.084282 17187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "e2ea791b04894d549c45bb6e3732764c"
format_stamp: "Formatted at 2026-09-28 21:51:55 on dist-test-slave-jrck"
server_key: "0a9f78c725e0bfaed1323aeb83e70ad1"
server_key_iv: "753ae2e6a51f3db7c8c99e8621bd99d4"
server_key_version: "encryptionkey@0"
I20260928 21:51:55.084635 17187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "e2ea791b04894d549c45bb6e3732764c"
format_stamp: "Formatted at 2026-09-28 21:51:55 on dist-test-slave-jrck"
server_key: "0a9f78c725e0bfaed1323aeb83e70ad1"
server_key_iv: "753ae2e6a51f3db7c8c99e8621bd99d4"
server_key_version: "encryptionkey@0"
I20260928 21:51:55.085812 17187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:51:55.086478 17202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.086627 17187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:51:55.086704 17187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "e2ea791b04894d549c45bb6e3732764c"
format_stamp: "Formatted at 2026-09-28 21:51:55 on dist-test-slave-jrck"
server_key: "0a9f78c725e0bfaed1323aeb83e70ad1"
server_key_iv: "753ae2e6a51f3db7c8c99e8621bd99d4"
server_key_version: "encryptionkey@0"
I20260928 21:51:55.086767 17187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:55.095121 17187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:55.096750 17187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:55.096868 17187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:55.097056 17187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:55.097329 17187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:55.097360 17187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.097385 17187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:55.097409 17187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.102690 17187 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:36507
I20260928 21:51:55.102743 17315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:36507 every 8 connection(s)
I20260928 21:51:55.103017 17187 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:51:55.104626 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 17187
I20260928 21:51:55.104709 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:51:55.104874 3075 external_mini_cluster.cc:1428] Setting key 20b552ed0fca9584fb1810c1a9cd20fb
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:55.111338 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.103609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:43623 (local address 127.3.0.254:36185)
0928 21:51:55.103654 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.103656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.103855 (+ 199us) server_negotiation.cc:407] Connection header received
0928 21:51:55.104340 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.104342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.104354 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.104407 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:55.105426 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.105913 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.106861 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.107044 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.107096 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.109038 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:55.109039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:55.109039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:55.109046 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:55.110117 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.110469 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.110470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.110470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.110520 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.110725 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.110726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.110726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.110834 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:55.110888 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.111136 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.111219 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:51:55.111460 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.103583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:43623)
0928 21:51:55.103739 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.103773 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.104241 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.104503 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.104508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.105120 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:55.105346 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.105351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.105990 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.105991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.106773 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.106776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.107108 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.107109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.107234 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.107634 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:55.107642 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:55.108954 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:55.110184 (+ 1230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:55.110190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:55.110193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:55.110403 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:55.110571 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:55.110572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:55.110573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:55.110673 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:55.110885 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:55.110891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:55.111023 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.111109 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.111255 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260928 21:51:55.111892 17316 heartbeater.cc:347] Connected to a master server at 127.3.0.254:36185
I20260928 21:51:55.111979 17316 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:55.112113 17316 heartbeater.cc:511] Master 127.3.0.254:36185 requested a full tablet report, sending...
I20260928 21:51:55.112440 16857 ts_manager.cc:194] Registered new tserver with Master: e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507)
I20260928 21:51:55.112828 16857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:43623
I20260928 21:51:55.116396 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:51:55.122763 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.117703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.3.0.254:36185)
0928 21:51:55.117757 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.117758 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.117885 (+ 127us) server_negotiation.cc:407] Connection header received
0928 21:51:55.117935 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.117935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.117943 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.117987 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:55.118464 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.118965 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.119642 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.119815 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.119861 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.120931 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:55.120932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:55.120933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:55.120939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:55.121973 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.122235 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.122236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.122236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.122278 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.122386 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.122386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.122386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.122509 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:55.122576 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.122648 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.122704 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:51:55.124074 16857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:51:55.125671 16857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:51:55.131385 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.127572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:34351 (local address 127.0.0.1:34652)
0928 21:51:55.127883 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.127927 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.127992 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.128400 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.128400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.128402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:55.128429 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.128430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.129327 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.129328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.130120 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.130122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.131038 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.131039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.131165 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.131183 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.131253 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.131298 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":63,"threads_started":1}
I20260928 21:51:55.131402 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.127683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.3.0.193:34351)
0928 21:51:55.127873 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.127878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.128075 (+ 197us) server_negotiation.cc:407] Connection header received
0928 21:51:55.128123 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.128125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.128141 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.128219 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:55.128519 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.129228 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.130200 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.130923 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.131039 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.131101 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.131279 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.131321 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":81,"threads_started":1}
I20260928 21:51:55.131610 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.127575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:36507 (local address 127.0.0.1:43524)
0928 21:51:55.127892 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.127930 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.127991 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.128318 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.128318 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.128320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:55.128357 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.128358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.129286 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.129287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.130500 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.130504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.131391 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.131392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.131453 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.131458 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.131506 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.131544 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":31,"thread_start_us":57,"threads_started":1}
I20260928 21:51:55.131870 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.127740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.3.0.195:36507)
0928 21:51:55.128112 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.128117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.128125 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:51:55.128153 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.128155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.128165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.128224 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:55.128426 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.129195 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.130564 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.131235 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.131304 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.131357 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.131757 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.131799 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":79,"threads_started":1}
I20260928 21:51:55.132097 16982 tablet_service.cc:1467] Processing CreateTablet for tablet a1c8a146b96a4e649b11dabe43a45efe (DEFAULT_TABLE table=test-table [id=2c9401ef55c4437ba0f953d47b199901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:55.132423 16982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c8a146b96a4e649b11dabe43a45efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:55.132700 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.127754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:43029 (local address 127.0.0.1:50852)
0928 21:51:55.127921 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.127930 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.127991 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.128228 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.128231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.128233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:55.128325 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.128327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.130533 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.130533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.131367 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.131369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.132488 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.132489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.132548 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.132552 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.132599 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.132636 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260928 21:51:55.132887 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.127872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.3.0.194:43029)
0928 21:51:55.128080 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.128085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.128094 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:51:55.128126 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.128128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.128141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.128216 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:55.129352 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.130426 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.131718 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.132391 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.132476 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.132533 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.132792 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.132831 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
I20260928 21:51:55.133023 17250 tablet_service.cc:1467] Processing CreateTablet for tablet a1c8a146b96a4e649b11dabe43a45efe (DEFAULT_TABLE table=test-table [id=2c9401ef55c4437ba0f953d47b199901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:55.133329 17250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c8a146b96a4e649b11dabe43a45efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:55.135874 17330 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Bootstrap starting.
I20260928 21:51:55.136256 17331 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: Bootstrap starting.
I20260928 21:51:55.136485 17330 tablet_bootstrap.cc:654] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:55.136933 17330 log.cc:824] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:55.136884 17116 tablet_service.cc:1467] Processing CreateTablet for tablet a1c8a146b96a4e649b11dabe43a45efe (DEFAULT_TABLE table=test-table [id=2c9401ef55c4437ba0f953d47b199901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:55.137140 17116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c8a146b96a4e649b11dabe43a45efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:55.137614 17330 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: No bootstrap required, opened a new log
I20260928 21:51:55.137674 17330 ts_tablet_manager.cc:1397] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:51:55.139230 17330 raft_consensus.cc:348] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.139369 17330 raft_consensus.cc:374] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:55.139410 17330 raft_consensus.cc:729] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b3ae98089b444aaddde508a09cc75b, State: Initialized, Role: FOLLOWER
I20260928 21:51:55.139520 17330 consensus_queue.cc:260] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.139715 17333 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Bootstrap starting.
I20260928 21:51:55.139786 17330 ts_tablet_manager.cc:1428] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:51:55.139806 17048 heartbeater.cc:503] Master 127.3.0.254:36185 was elected leader, sending a full tablet report...
I20260928 21:51:55.140242 17331 tablet_bootstrap.cc:654] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:55.140499 17333 tablet_bootstrap.cc:654] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:55.140775 17331 log.cc:824] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:55.141041 17333 log.cc:824] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:55.141430 17331 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: No bootstrap required, opened a new log
I20260928 21:51:55.141482 17331 ts_tablet_manager.cc:1397] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260928 21:51:55.141681 17333 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: No bootstrap required, opened a new log
I20260928 21:51:55.141746 17333 ts_tablet_manager.cc:1397] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 21:51:55.142786 17331 raft_consensus.cc:348] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.142912 17331 raft_consensus.cc:374] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:55.142942 17331 raft_consensus.cc:729] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ea791b04894d549c45bb6e3732764c, State: Initialized, Role: FOLLOWER
I20260928 21:51:55.143074 17331 consensus_queue.cc:260] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.143035 17333 raft_consensus.cc:348] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.143198 17333 raft_consensus.cc:374] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:55.143239 17333 raft_consensus.cc:729] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89225913bb0147d59cfe79b17ade11d0, State: Initialized, Role: FOLLOWER
I20260928 21:51:55.143319 17331 ts_tablet_manager.cc:1428] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 21:51:55.143356 17333 consensus_queue.cc:260] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.143438 17316 heartbeater.cc:503] Master 127.3.0.254:36185 was elected leader, sending a full tablet report...
I20260928 21:51:55.143611 17333 ts_tablet_manager.cc:1428] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:51:55.143620 17182 heartbeater.cc:503] Master 127.3.0.254:36185 was elected leader, sending a full tablet report...
W20260928 21:51:55.211182 17183 tablet.cc:2369] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:51:55.227800 17337 raft_consensus.cc:482] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:51:55.227957 17337 raft_consensus.cc:504] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.228317 17337 leader_election.cc:288] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20b3ae98089b444aaddde508a09cc75b (127.3.0.193:34351), 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029)
I20260928 21:51:55.231174 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.228598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:57343 (local address 127.3.0.194:43029)
0928 21:51:55.228648 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.228650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.228852 (+ 202us) server_negotiation.cc:407] Connection header received
0928 21:51:55.228881 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.228883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.228896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.228949 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:55.229095 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.229567 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.230482 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.230878 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.230939 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.230948 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.231078 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.231117 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:51:55.231163 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.228560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:52019 (local address 127.3.0.193:34351)
0928 21:51:55.228602 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.228604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.228613 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:51:55.228702 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.228704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.228715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.228770 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:55.228946 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.229484 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.230404 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.230811 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.230879 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.230890 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.231046 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.231089 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:51:55.231164 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.228501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:34351 (local address 127.3.0.195:52019)
0928 21:51:55.228564 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.228576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.228627 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.228785 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.228787 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.228789 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:55.228861 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.228863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.229559 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.229560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.230319 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.230321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.230901 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.230902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.230982 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.230985 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.231029 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.231068 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:51:55.231189 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.228509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:43029 (local address 127.3.0.195:57343)
0928 21:51:55.228791 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.228799 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.228846 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.228960 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.228961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.228962 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:55.229032 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.229033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.229646 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.229647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.230400 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.230403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.230956 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.230957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.231011 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.231015 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.231062 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.231097 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":113,"thread_start_us":114,"threads_started":1}
I20260928 21:51:55.231549 17136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c8a146b96a4e649b11dabe43a45efe" candidate_uuid: "e2ea791b04894d549c45bb6e3732764c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89225913bb0147d59cfe79b17ade11d0" is_pre_election: true
I20260928 21:51:55.231600 17002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c8a146b96a4e649b11dabe43a45efe" candidate_uuid: "e2ea791b04894d549c45bb6e3732764c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b3ae98089b444aaddde508a09cc75b" is_pre_election: true
I20260928 21:51:55.231700 17136 raft_consensus.cc:2454] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ea791b04894d549c45bb6e3732764c in term 0.
I20260928 21:51:55.231751 17002 raft_consensus.cc:2454] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2ea791b04894d549c45bb6e3732764c in term 0.
I20260928 21:51:55.231943 17206 leader_election.cc:302] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89225913bb0147d59cfe79b17ade11d0, e2ea791b04894d549c45bb6e3732764c; no voters:
I20260928 21:51:55.232059 17337 raft_consensus.cc:2781] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:51:55.232123 17337 raft_consensus.cc:482] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:51:55.232141 17337 raft_consensus.cc:3037] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:55.232892 17337 raft_consensus.cc:504] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.233011 17337 leader_election.cc:288] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [CANDIDATE]: Term 1 election: Requested vote from peers 20b3ae98089b444aaddde508a09cc75b (127.3.0.193:34351), 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029)
I20260928 21:51:55.233186 17002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c8a146b96a4e649b11dabe43a45efe" candidate_uuid: "e2ea791b04894d549c45bb6e3732764c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b3ae98089b444aaddde508a09cc75b"
I20260928 21:51:55.233214 17136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c8a146b96a4e649b11dabe43a45efe" candidate_uuid: "e2ea791b04894d549c45bb6e3732764c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89225913bb0147d59cfe79b17ade11d0"
I20260928 21:51:55.233270 17002 raft_consensus.cc:3037] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:55.233285 17136 raft_consensus.cc:3037] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:55.233934 17136 raft_consensus.cc:2454] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ea791b04894d549c45bb6e3732764c in term 1.
I20260928 21:51:55.234025 17002 raft_consensus.cc:2454] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2ea791b04894d549c45bb6e3732764c in term 1.
I20260928 21:51:55.234092 17206 leader_election.cc:302] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89225913bb0147d59cfe79b17ade11d0, e2ea791b04894d549c45bb6e3732764c; no voters:
I20260928 21:51:55.234174 17337 raft_consensus.cc:2781] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:51:55.234308 17337 raft_consensus.cc:686] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 1 LEADER]: Becoming Leader. State: Replica: e2ea791b04894d549c45bb6e3732764c, State: Running, Role: LEADER
I20260928 21:51:55.234400 17337 consensus_queue.cc:237] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.235219 16857 catalog_manager.cc:5495] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c reported cstate change: term changed from 0 to 1, leader changed from <none> to e2ea791b04894d549c45bb6e3732764c (127.3.0.195). New cstate: current_term: 1 leader_uuid: "e2ea791b04894d549c45bb6e3732764c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } health_report { overall_health: UNKNOWN } } }
I20260928 21:51:55.244783 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.242624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.3.0.195:36507)
0928 21:51:55.242660 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.242662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.242669 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:51:55.242739 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.242740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.242758 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.242812 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:51:55.242935 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.243386 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.244095 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.244276 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.244326 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.244499 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:51:55.244591 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:51:55.244660 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.244679 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.244700 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:51:55.246418 17002 raft_consensus.cc:1261] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Refusing update from remote peer e2ea791b04894d549c45bb6e3732764c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:51:55.246428 17136 raft_consensus.cc:1261] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Refusing update from remote peer e2ea791b04894d549c45bb6e3732764c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:51:55.246737 17340 consensus_queue.cc:1035] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:51:55.246846 17337 consensus_queue.cc:1035] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [LEADER]: Connected to new peer: Peer: permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:51:55.249775 17345 mvcc.cc:205] Tried to move back new op lower bound from 7334429963247079424 to 7334429963200946176. Current Snapshot: MvccSnapshot[applied={T|T < 7334429963247079424 or (T in {7334429963247079424})}]
I20260928 21:51:55.251066 16856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:51:55.251142 16856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:51:55.251770 16856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Sending DeleteTablet for 3 replicas of tablet a1c8a146b96a4e649b11dabe43a45efe
I20260928 21:51:55.252043 17250 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c8a146b96a4e649b11dabe43a45efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:55 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:43524
I20260928 21:51:55.252082 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 16919
I20260928 21:51:55.252243 17116 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c8a146b96a4e649b11dabe43a45efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:55 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:50852
I20260928 21:51:55.252446 17352 tablet_replica.cc:330] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: stopping tablet replica
I20260928 21:51:55.252552 17352 raft_consensus.cc:2229] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:51:55.252640 17352 raft_consensus.cc:2258] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:51:55.253000 17352 ts_tablet_manager.cc:1905] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:51:55.254518 17352 ts_tablet_manager.cc:1918] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:51:55.254590 17352 log.cc:1196] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/a1c8a146b96a4e649b11dabe43a45efe
I20260928 21:51:55.254849 17352 ts_tablet_manager.cc:1939] T a1c8a146b96a4e649b11dabe43a45efe P e2ea791b04894d549c45bb6e3732764c: Deleting consensus metadata
I20260928 21:51:55.255323 16844 catalog_manager.cc:4841] TS e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507): tablet a1c8a146b96a4e649b11dabe43a45efe (table test-table [id=2c9401ef55c4437ba0f953d47b199901]) successfully deleted
W20260928 21:51:55.257262 16841 connection.cc:535] client connection to 127.3.0.193:34351 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:51:55.257361 16841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:51:55.257781 17353 tablet_replica.cc:330] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: stopping tablet replica
I20260928 21:51:55.257804 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.257671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:34351 (local address 127.0.0.1:34656)
0928 21:51:55.257725 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.257770 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:34351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 21:51:55.257895 17353 raft_consensus.cc:2229] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 21:51:55.257922 16841 catalog_manager.cc:4584] TS 20b3ae98089b444aaddde508a09cc75b (127.3.0.193:34351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1c8a146b96a4e649b11dabe43a45efe: Network error: Client connection negotiation failed: client connection to 127.3.0.193:34351: connect: Connection refused (error 111)
I20260928 21:51:55.258014 17353 raft_consensus.cc:2258] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:51:55.258194 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 17053
I20260928 21:51:55.263983 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.263877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:43029 (local address 127.0.0.1:50864)
0928 21:51:55.263927 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.263953 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.194:43029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:51:55.263999 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 17187
I20260928 21:51:55.268257 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 16825
I20260928 21:51:55.272723 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:36185
--webserver_interface=127.3.0.254
--webserver_port=37581
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:55.351424 17355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:55.351630 17355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:55.351651 17355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:55.352808 17355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:51:55.352845 17355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:55.352859 17355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:55.352869 17355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:51:55.352880 17355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:51:55.354332 17355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:55.354560 17355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:55.354773 17355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:55.362960 17361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:55.362963 17362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:55.363016 17355 server_base.cc:1034] running on GCE node
W20260928 21:51:55.362962 17364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:55.363372 17355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:55.363683 17355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:55.364830 17355 hybrid_clock.cc:648] HybridClock initialized: now 1790632315364814 us; error 33 us; skew 500 ppm
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:55.367184 17355 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:51:55.367632 17355 webserver.cc:460] Webserver started at http://127.3.0.254:37581/ using document root <none> and password file <none>
I20260928 21:51:55.367842 17355 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:55.367913 17355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:55.369263 17355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:55.369840 17371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.369989 17355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:55.370072 17355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "40d07be0c92a49b58eceb8bb2e31d6a5"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "4bdd7a90df6a3421d9bffaa6882c2973"
server_key_iv: "0cd5837a15ff0dad0ed3e2816798dadc"
server_key_version: "encryptionkey@0"
I20260928 21:51:55.370462 17355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:55.381659 17355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:55.383376 17355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:55.383507 17355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:55.387281 17355 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:36185
I20260928 21:51:55.387320 17423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:36185 every 8 connection(s)
I20260928 21:51:55.387671 17355 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:51:55.389225 17424 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:51:55.389870 17424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Bootstrap starting.
I20260928 21:51:55.397017 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 17355
I20260928 21:51:55.397266 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:34351
--local_ip_for_outbound_sockets=127.3.0.193
--tserver_master_addrs=127.3.0.254:36185
--webserver_port=39613
--webserver_interface=127.3.0.193
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:51:55.397791 17424 log.cc:824] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:55.400074 17424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:51:55.400326 17424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Bootstrap complete.
I20260928 21:51:55.402432 17424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:55.402648 17424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d07be0c92a49b58eceb8bb2e31d6a5, State: Initialized, Role: FOLLOWER
I20260928 21:51:55.402786 17424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:55.402848 17424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:51:55.402892 17424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:51:55.402940 17424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:51:55.403900 17424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:55.403981 17424 leader_election.cc:302] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40d07be0c92a49b58eceb8bb2e31d6a5; no voters:
I20260928 21:51:55.404122 17424 leader_election.cc:288] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:51:55.404167 17428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:51:55.404274 17428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 2 LEADER]: Becoming Leader. State: Replica: 40d07be0c92a49b58eceb8bb2e31d6a5, State: Running, Role: LEADER
I20260928 21:51:55.404372 17424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:51:55.404364 17428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:55.404703 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d07be0c92a49b58eceb8bb2e31d6a5. Latest consensus state: current_term: 2 leader_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } } }
I20260928 21:51:55.404785 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:55.404997 17430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } } }
I20260928 21:51:55.405107 17430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:55.405463 17434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:51:55.406211 17434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:51:55.406308 17434 catalog_manager.cc:1259] Loaded cluster ID: b8d3554a21504e99bf0fc91df0fc6b05
I20260928 21:51:55.406344 17434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:51:55.406562 17434 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 21:51:55.411152 17434 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 21:51:55.413513 17434 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:51:55.413714 17434 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Loaded TSK: 0
I20260928 21:51:55.413967 17434 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:51:55.477078 17427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:55.477272 17427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:55.477303 17427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:55.478402 17427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:55.478442 17427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:55.478477 17427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:51:55.479909 17427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:34351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=39613
--tserver_master_addrs=127.3.0.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:55.480136 17427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:55.480338 17427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:55.485014 17451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:55.485167 17454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:55.485014 17452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:55.485637 17427 server_base.cc:1034] running on GCE node
I20260928 21:51:55.485827 17427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:55.486042 17427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:55.487178 17427 hybrid_clock.cc:648] HybridClock initialized: now 1790632315487165 us; error 31 us; skew 500 ppm
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:55.489687 17427 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:51:55.490127 17427 webserver.cc:460] Webserver started at http://127.3.0.193:39613/ using document root <none> and password file <none>
I20260928 21:51:55.490332 17427 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:55.490392 17427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:55.491657 17427 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:55.492276 17461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.492434 17427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:55.492550 17427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "20b3ae98089b444aaddde508a09cc75b"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "daf3149c7407a562b74937ccffd4df44"
server_key_iv: "666a3d800480231728c63cbc8c72d617"
server_key_version: "encryptionkey@0"
I20260928 21:51:55.492995 17427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:55.508352 17427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:55.510586 17427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:55.510735 17427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:55.510998 17427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:55.511488 17468 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:51:55.512948 17427 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:51:55.513007 17427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 21:51:55.513041 17427 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:51:55.513892 17427 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:51:55.513938 17427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 21:51:55.513995 17468 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Bootstrap starting.
I20260928 21:51:55.520965 17427 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:34351
I20260928 21:51:55.521144 17575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:34351 every 8 connection(s)
I20260928 21:51:55.521374 17427 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:51:55.522133 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 17427
I20260928 21:51:55.522399 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:43029
--local_ip_for_outbound_sockets=127.3.0.194
--tserver_master_addrs=127.3.0.254:36185
--webserver_port=43353
--webserver_interface=127.3.0.194
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:51:55.523674 17468 log.cc:824] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:55.525981 17468 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:51:55.526225 17468 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Bootstrap complete.
I20260928 21:51:55.526975 17468 ts_tablet_manager.cc:1397] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260928 21:51:55.528149 17468 raft_consensus.cc:348] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.528398 17468 raft_consensus.cc:729] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b3ae98089b444aaddde508a09cc75b, State: Initialized, Role: FOLLOWER
I20260928 21:51:55.528604 17468 consensus_queue.cc:260] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.528873 17468 ts_tablet_manager.cc:1428] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:55.539959 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.529155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:37659 (local address 127.3.0.254:36185)
0928 21:51:55.529334 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.529357 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.529466 (+ 109us) server_negotiation.cc:407] Connection header received
0928 21:51:55.529698 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.529705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.529762 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.529907 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:55.530688 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.532088 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.533527 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.534140 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.534257 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.536863 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:55.536868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:55.536868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:55.536880 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:55.538395 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.538846 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.538848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.538848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.538924 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.539165 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.539166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.539166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.539359 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:55.539415 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.539703 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.539817 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260928 21:51:55.540130 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.529099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:37659)
0928 21:51:55.529272 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.529321 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.529504 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.529953 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.529957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.530308 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:55.530563 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.530569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.532186 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.532187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.533441 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.533443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.534244 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.534246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.534382 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.534980 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:55.534990 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:55.536752 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:55.538469 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:55.538476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:55.538478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:55.538776 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:55.538976 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:55.538977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:55.538978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:55.539099 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:55.539410 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:55.539414 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:55.539566 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.539669 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.539830 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260928 21:51:55.540856 17576 heartbeater.cc:347] Connected to a master server at 127.3.0.254:36185
I20260928 21:51:55.540946 17576 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:55.541109 17576 heartbeater.cc:511] Master 127.3.0.254:36185 requested a full tablet report, sending...
I20260928 21:51:55.541823 17388 ts_manager.cc:194] Registered new tserver with Master: 20b3ae98089b444aaddde508a09cc75b (127.3.0.193:34351)
I20260928 21:51:55.542991 17388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:37659
I20260928 21:51:55.543828 17576 heartbeater.cc:503] Master 127.3.0.254:36185 was elected leader, sending a full tablet report...
I20260928 21:51:55.546515 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.542393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:34351 (local address 127.0.0.1:34658)
0928 21:51:55.543288 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.543319 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.543390 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.543811 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.543814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.543817 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:55.543877 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.543879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.544788 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.544789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.545576 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.545578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.546234 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.546235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.546324 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.546335 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.546392 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.546446 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":107,"threads_started":1}
I20260928 21:51:55.546525 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.542457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.3.0.193:34351)
0928 21:51:55.543511 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.543515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.543524 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:51:55.543554 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.543556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.543707 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.543778 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:55.543964 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.544711 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.545652 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.546142 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.546214 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.546268 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.546413 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.546450 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":71,"threads_started":1}
I20260928 21:51:55.546938 17492 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c8a146b96a4e649b11dabe43a45efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:55 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:34658
I20260928 21:51:55.547473 17585 tablet_replica.cc:330] stopping tablet replica
I20260928 21:51:55.547580 17585 raft_consensus.cc:2229] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:51:55.547643 17585 raft_consensus.cc:2258] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:51:55.547936 17585 ts_tablet_manager.cc:1905] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:51:55.550079 17585 ts_tablet_manager.cc:1918] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:51:55.550153 17585 log.cc:1196] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/a1c8a146b96a4e649b11dabe43a45efe
I20260928 21:51:55.550366 17585 ts_tablet_manager.cc:1939] T a1c8a146b96a4e649b11dabe43a45efe P 20b3ae98089b444aaddde508a09cc75b: Deleting consensus metadata
I20260928 21:51:55.550897 17372 catalog_manager.cc:4841] TS 20b3ae98089b444aaddde508a09cc75b (127.3.0.193:34351): tablet a1c8a146b96a4e649b11dabe43a45efe (table test-table [id=2c9401ef55c4437ba0f953d47b199901]) successfully deleted
W20260928 21:51:55.610947 17578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:55.611145 17578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:55.611167 17578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:55.612350 17578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:55.612392 17578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:55.612421 17578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:51:55.613885 17578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:43029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=43353
--tserver_master_addrs=127.3.0.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:55.614099 17578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:55.614307 17578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:55.619095 17591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:55.619088 17593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:55.619228 17578 server_base.cc:1034] running on GCE node
W20260928 21:51:55.619088 17590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:55.619490 17578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:55.619694 17578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:55.620851 17578 hybrid_clock.cc:648] HybridClock initialized: now 1790632315620832 us; error 42 us; skew 500 ppm
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:55.623181 17578 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:51:55.623586 17578 webserver.cc:460] Webserver started at http://127.3.0.194:43353/ using document root <none> and password file <none>
I20260928 21:51:55.623801 17578 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:55.623863 17578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:55.625196 17578 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:51:55.625792 17600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.625938 17578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:55.626027 17578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "89225913bb0147d59cfe79b17ade11d0"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "ef212f4e9f2b9802d9585a22b93903f9"
server_key_iv: "29b022a9e3adeab40705f30dffc79a1b"
server_key_version: "encryptionkey@0"
I20260928 21:51:55.626359 17578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:55.639671 17578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:55.641489 17578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:55.641619 17578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:55.641831 17578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:55.642227 17607 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:51:55.643620 17578 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:51:55.643669 17578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 21:51:55.643702 17578 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:51:55.644336 17578 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:51:55.644367 17578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 21:51:55.644420 17607 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Bootstrap starting.
I20260928 21:51:55.649560 17578 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:43029
I20260928 21:51:55.649645 17714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:43029 every 8 connection(s)
I20260928 21:51:55.649935 17578 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:51:55.652206 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 17578
I20260928 21:51:55.652493 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:36507
--local_ip_for_outbound_sockets=127.3.0.195
--tserver_master_addrs=127.3.0.254:36185
--webserver_port=42419
--webserver_interface=127.3.0.195
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:51:55.654069 17607 log.cc:824] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:55.657078 17607 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:51:55.657343 17607 tablet_bootstrap.cc:492] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Bootstrap complete.
I20260928 21:51:55.658257 17607 ts_tablet_manager.cc:1397] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260928 21:51:55.659358 17607 raft_consensus.cc:348] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.659699 17607 raft_consensus.cc:729] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89225913bb0147d59cfe79b17ade11d0, State: Initialized, Role: FOLLOWER
I20260928 21:51:55.659847 17607 consensus_queue.cc:260] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } }
I20260928 21:51:55.660106 17607 ts_tablet_manager.cc:1428] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 21:51:55.660346 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.650626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:55331 (local address 127.3.0.254:36185)
0928 21:51:55.650689 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.650691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.650848 (+ 157us) server_negotiation.cc:407] Connection header received
0928 21:51:55.651288 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.651289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.651301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.651387 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:55.651947 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.652769 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.654153 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.654458 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.654536 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.657138 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:55.657139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:55.657140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:55.657149 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:55.658759 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.659239 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.659240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.659241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.659301 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.659582 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.659583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.659583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.659717 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:55.659791 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.660186 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.660273 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:51:55.660529 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.650548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:55331)
0928 21:51:55.650713 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.650753 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.651192 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.651430 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.651434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.651669 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:55.651854 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.651858 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.652918 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.652919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.654066 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.654069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.654570 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.654571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.654735 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.655286 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:55.655295 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:55.657017 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:55.658852 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:55.658858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:55.658860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:55.659159 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:55.659368 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:55.659369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:55.659370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:55.659514 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:55.659823 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:55.659830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:55.660001 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.660119 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.660292 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260928 21:51:55.660980 17715 heartbeater.cc:347] Connected to a master server at 127.3.0.254:36185
I20260928 21:51:55.661075 17715 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:55.661234 17715 heartbeater.cc:511] Master 127.3.0.254:36185 requested a full tablet report, sending...
I20260928 21:51:55.661727 17388 ts_manager.cc:194] Registered new tserver with Master: 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029)
I20260928 21:51:55.662204 17388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:55331
I20260928 21:51:55.663002 17715 heartbeater.cc:503] Master 127.3.0.254:36185 was elected leader, sending a full tablet report...
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.194@KRBTEST.COM
I20260928 21:51:55.669312 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.662337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.3.0.194:43029)
0928 21:51:55.662477 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.662482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.662491 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:51:55.662525 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.662526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.662540 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.662603 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:55.663015 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.663782 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.664415 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.664750 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.664808 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.666879 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:55.666885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:55.666885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:55.666891 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:55.668177 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.668632 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.668634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.668635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.668685 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.668858 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.668859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.668859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.668976 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:55.669032 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.669182 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.669250 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260928 21:51:55.669348 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.662019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:43029 (local address 127.0.0.1:50878)
0928 21:51:55.662061 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.662072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.662110 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.662614 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.662616 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.662908 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:55.662941 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.662943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.663854 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.663856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.664348 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.664349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.664817 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.664818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.664876 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.665339 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:55.665343 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:55.666809 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:55.668251 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:55.668259 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:55.668262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:55.668556 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:55.668735 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:55.668736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:55.668737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:55.668809 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:55.669025 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:55.669030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:55.669087 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.669158 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.669251 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 21:51:55.669765 17649 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c8a146b96a4e649b11dabe43a45efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:55 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:50878
I20260928 21:51:55.670226 17722 tablet_replica.cc:330] stopping tablet replica
I20260928 21:51:55.670315 17722 raft_consensus.cc:2229] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:51:55.670394 17722 raft_consensus.cc:2258] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:51:55.670655 17722 ts_tablet_manager.cc:1905] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:51:55.672616 17722 ts_tablet_manager.cc:1918] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:51:55.672688 17722 log.cc:1196] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/a1c8a146b96a4e649b11dabe43a45efe
I20260928 21:51:55.672874 17722 ts_tablet_manager.cc:1939] T a1c8a146b96a4e649b11dabe43a45efe P 89225913bb0147d59cfe79b17ade11d0: Deleting consensus metadata
I20260928 21:51:55.673312 17375 catalog_manager.cc:4841] TS 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029): tablet a1c8a146b96a4e649b11dabe43a45efe (table test-table [id=2c9401ef55c4437ba0f953d47b199901]) successfully deleted
W20260928 21:51:55.735523 17718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:55.735728 17718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:55.735749 17718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:55.736876 17718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:55.736910 17718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:55.736941 17718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:51:55.738410 17718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:36507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=42419
--tserver_master_addrs=127.3.0.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:55.738602 17718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:55.738777 17718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:55.743374 17727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:55.743388 17728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:55.743417 17718 server_base.cc:1034] running on GCE node
W20260928 21:51:55.743392 17730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:55.743716 17718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:55.743932 17718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:55.745069 17718 hybrid_clock.cc:648] HybridClock initialized: now 1790632315745053 us; error 29 us; skew 500 ppm
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:55.747470 17718 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:51:55.747879 17718 webserver.cc:460] Webserver started at http://127.3.0.195:42419/ using document root <none> and password file <none>
I20260928 21:51:55.748077 17718 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:55.748137 17718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:55.749428 17718 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:55.749945 17737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.750088 17718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:55.750165 17718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "e2ea791b04894d549c45bb6e3732764c"
format_stamp: "Formatted at 2026-09-28 21:51:55 on dist-test-slave-jrck"
server_key: "0a9f78c725e0bfaed1323aeb83e70ad1"
server_key_iv: "753ae2e6a51f3db7c8c99e8621bd99d4"
server_key_version: "encryptionkey@0"
I20260928 21:51:55.750486 17718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:55.777254 17718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:55.779058 17718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:55.779213 17718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:55.779428 17718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:55.779721 17718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:55.779754 17718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.779783 17718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:55.779800 17718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.785231 17718 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:36507
I20260928 21:51:55.785305 17850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:36507 every 8 connection(s)
I20260928 21:51:55.785611 17718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:51:55.787755 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 17718
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:51:55.794319 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.788112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.3.0.254:36185)
0928 21:51:55.788225 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.788227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.788290 (+ 63us) server_negotiation.cc:407] Connection header received
0928 21:51:55.788354 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.788355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.788366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.788419 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:55.788996 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.789445 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.790279 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.790455 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.790509 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.791608 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:55.791609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:55.791610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:55.791616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:55.793263 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.793569 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.793569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.793570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.793624 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.793787 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.793788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.793788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.793917 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:55.794092 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.794132 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.794212 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260928 21:51:55.794894 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.786275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:51585 (local address 127.3.0.254:36185)
0928 21:51:55.786351 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.786353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.786563 (+ 210us) server_negotiation.cc:407] Connection header received
0928 21:51:55.787030 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.787031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.787041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.787092 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:55.788144 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.788698 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.789875 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.790050 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.790097 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.792244 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:55.792245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:55.792246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:55.792253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:55.793562 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.793893 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.793894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.793894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.793936 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:55.794224 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:55.794225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:55.794225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:55.794344 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:55.794383 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.794732 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.794810 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:51:55.795077 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.786274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:51585)
0928 21:51:55.786429 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.786467 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.786950 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.787160 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.787163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.787787 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:55.788057 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.788063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.788787 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.788788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.789795 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.789797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.790115 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.790116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.790269 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.790757 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:55.790769 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:55.792156 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:55.793622 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:55.793628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:55.793630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:55.793834 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:55.794010 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:55.794011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:55.794011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:55.794146 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:55.794430 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:55.794436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:55.794588 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.794696 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.794858 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260928 21:51:55.795495 17851 heartbeater.cc:347] Connected to a master server at 127.3.0.254:36185
I20260928 21:51:55.795584 17851 heartbeater.cc:464] Registering TS with master...
I20260928 21:51:55.795749 17851 heartbeater.cc:511] Master 127.3.0.254:36185 requested a full tablet report, sending...
I20260928 21:51:55.796046 17388 ts_manager.cc:194] Registered new tserver with Master: e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507)
I20260928 21:51:55.796383 17388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:51585
I20260928 21:51:55.805935 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:51:55.808596 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.806278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.3.0.254:36185)
0928 21:51:55.806326 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.806328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.806430 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:51:55.806494 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.806494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.806531 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.806591 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:51:55.806686 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.807160 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.807878 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.808059 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.808101 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.808258 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:51:55.808355 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:51:55.808413 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.808523 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.808538 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:51:55.808866 17388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:51:55.810372 17388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:51:55.812355 17492 tablet_service.cc:1467] Processing CreateTablet for tablet aae8b894852f4df3b485ebb13a00e909 (DEFAULT_TABLE table=test-table [id=4b99a566b41d4451ae46ba126c9e72a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:55.812354 17649 tablet_service.cc:1467] Processing CreateTablet for tablet aae8b894852f4df3b485ebb13a00e909 (DEFAULT_TABLE table=test-table [id=4b99a566b41d4451ae46ba126c9e72a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:55.812566 17492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae8b894852f4df3b485ebb13a00e909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:55.812572 17649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae8b894852f4df3b485ebb13a00e909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:55.813796 17468 tablet_bootstrap.cc:492] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: Bootstrap starting.
I20260928 21:51:55.814050 17607 tablet_bootstrap.cc:492] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0: Bootstrap starting.
I20260928 21:51:55.814201 17468 tablet_bootstrap.cc:654] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:55.814318 17607 tablet_bootstrap.cc:654] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:55.814836 17468 tablet_bootstrap.cc:492] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: No bootstrap required, opened a new log
I20260928 21:51:55.814877 17468 ts_tablet_manager.cc:1397] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:55.814999 17468 raft_consensus.cc:348] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:55.815063 17468 raft_consensus.cc:374] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:55.815094 17468 raft_consensus.cc:729] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b3ae98089b444aaddde508a09cc75b, State: Initialized, Role: FOLLOWER
I20260928 21:51:55.815145 17468 consensus_queue.cc:260] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:55.815231 17468 ts_tablet_manager.cc:1428] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.815549 17607 tablet_bootstrap.cc:492] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0: No bootstrap required, opened a new log
I20260928 21:51:55.815594 17607 ts_tablet_manager.cc:1397] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 21:51:55.815745 17607 raft_consensus.cc:348] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:55.815806 17607 raft_consensus.cc:374] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:55.815851 17607 raft_consensus.cc:729] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89225913bb0147d59cfe79b17ade11d0, State: Initialized, Role: FOLLOWER
I20260928 21:51:55.815850 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.812044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:36507 (local address 127.0.0.1:43542)
0928 21:51:55.812073 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.812082 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.812113 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.812544 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.812545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.812547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:55.812597 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.812598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.813880 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.813881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.814765 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.814768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.815624 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.815625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.815690 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.815694 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.815722 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.815766 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 21:51:55.815894 17607 consensus_queue.cc:260] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:55.815976 17607 ts_tablet_manager.cc:1428] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.816222 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.812136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.3.0.195:36507)
0928 21:51:55.812329 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.812335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.812345 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:51:55.812382 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.812384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.812400 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.812668 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:55.812741 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.813743 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.814852 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.815532 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.816007 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.816079 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.816105 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.816147 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":106,"threads_started":1}
I20260928 21:51:55.816924 17785 tablet_service.cc:1467] Processing CreateTablet for tablet aae8b894852f4df3b485ebb13a00e909 (DEFAULT_TABLE table=test-table [id=4b99a566b41d4451ae46ba126c9e72a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:51:55.817230 17785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae8b894852f4df3b485ebb13a00e909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:51:55.819912 17860 tablet_bootstrap.cc:492] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: Bootstrap starting.
I20260928 21:51:55.820490 17860 tablet_bootstrap.cc:654] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: Neither blocks nor log segments found. Creating new log.
I20260928 21:51:55.820950 17860 log.cc:824] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:55.821569 17860 tablet_bootstrap.cc:492] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: No bootstrap required, opened a new log
I20260928 21:51:55.821628 17860 ts_tablet_manager.cc:1397] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:51:55.822954 17860 raft_consensus.cc:348] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:55.823092 17860 raft_consensus.cc:374] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:51:55.823122 17860 raft_consensus.cc:729] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ea791b04894d549c45bb6e3732764c, State: Initialized, Role: FOLLOWER
I20260928 21:51:55.823235 17860 consensus_queue.cc:260] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:55.823470 17860 ts_tablet_manager.cc:1428] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:51:55.823509 17851 heartbeater.cc:503] Master 127.3.0.254:36185 was elected leader, sending a full tablet report...
I20260928 21:51:55.831367 17720 raft_consensus.cc:482] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:51:55.831483 17720 raft_consensus.cc:504] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:55.831765 17720 leader_election.cc:288] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507), 20b3ae98089b444aaddde508a09cc75b (127.3.0.193:34351)
I20260928 21:51:55.834337 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.831894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:36507 (local address 127.3.0.194:58287)
0928 21:51:55.831942 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.831951 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.831996 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.832151 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.832152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.832153 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:55.832216 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.832218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.832847 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.832848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.833563 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.833565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.834075 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.834076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.834168 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.834184 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.834230 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.834272 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:51:55.834525 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.831939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:58287 (local address 127.3.0.195:36507)
0928 21:51:55.831987 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.831989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.831996 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:51:55.832071 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.832073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.832084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.832134 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:55.832266 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.832783 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.833641 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.834009 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.834068 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.834077 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.834433 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.834473 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:51:55.834570 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.831922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:34351 (local address 127.3.0.194:56733)
0928 21:51:55.832061 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.832070 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:55.832110 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:55.832279 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:55.832280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:55.832282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:51:55.832355 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.832357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.832958 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.832959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.833689 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:55.833691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.834260 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.834261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:55.834328 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.834332 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:51:55.834383 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:51:55.834482 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260928 21:51:55.834597 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.832140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:56733 (local address 127.3.0.193:34351)
0928 21:51:55.832180 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.832182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.832189 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:51:55.832208 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.832208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.832217 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.832257 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:51:55.832409 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.832893 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.833777 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.834147 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.834213 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.834223 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.834498 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.834542 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:51:55.834803 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae8b894852f4df3b485ebb13a00e909" candidate_uuid: "89225913bb0147d59cfe79b17ade11d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ea791b04894d549c45bb6e3732764c" is_pre_election: true
I20260928 21:51:55.834846 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae8b894852f4df3b485ebb13a00e909" candidate_uuid: "89225913bb0147d59cfe79b17ade11d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b3ae98089b444aaddde508a09cc75b" is_pre_election: true
I20260928 21:51:55.834915 17805 raft_consensus.cc:2454] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89225913bb0147d59cfe79b17ade11d0 in term 0.
I20260928 21:51:55.834995 17530 raft_consensus.cc:2454] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89225913bb0147d59cfe79b17ade11d0 in term 0.
I20260928 21:51:55.835107 17604 leader_election.cc:302] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89225913bb0147d59cfe79b17ade11d0, e2ea791b04894d549c45bb6e3732764c; no voters:
I20260928 21:51:55.835223 17720 raft_consensus.cc:2781] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:51:55.835276 17720 raft_consensus.cc:482] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:51:55.835295 17720 raft_consensus.cc:3037] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:55.835884 17720 raft_consensus.cc:504] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:55.835989 17720 leader_election.cc:288] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [CANDIDATE]: Term 1 election: Requested vote from peers e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507), 20b3ae98089b444aaddde508a09cc75b (127.3.0.193:34351)
I20260928 21:51:55.836122 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae8b894852f4df3b485ebb13a00e909" candidate_uuid: "89225913bb0147d59cfe79b17ade11d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b3ae98089b444aaddde508a09cc75b"
I20260928 21:51:55.836150 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae8b894852f4df3b485ebb13a00e909" candidate_uuid: "89225913bb0147d59cfe79b17ade11d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ea791b04894d549c45bb6e3732764c"
I20260928 21:51:55.836197 17530 raft_consensus.cc:3037] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:55.836210 17805 raft_consensus.cc:3037] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:51:55.836879 17805 raft_consensus.cc:2454] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89225913bb0147d59cfe79b17ade11d0 in term 1.
I20260928 21:51:55.837035 17604 leader_election.cc:302] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89225913bb0147d59cfe79b17ade11d0, e2ea791b04894d549c45bb6e3732764c; no voters:
I20260928 21:51:55.837111 17720 raft_consensus.cc:2781] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:51:55.837255 17530 raft_consensus.cc:2454] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89225913bb0147d59cfe79b17ade11d0 in term 1.
I20260928 21:51:55.837272 17720 raft_consensus.cc:686] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 1 LEADER]: Becoming Leader. State: Replica: 89225913bb0147d59cfe79b17ade11d0, State: Running, Role: LEADER
I20260928 21:51:55.837332 17720 consensus_queue.cc:237] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:55.838016 17388 catalog_manager.cc:5495] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194). New cstate: current_term: 1 leader_uuid: "89225913bb0147d59cfe79b17ade11d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } health_report { overall_health: UNKNOWN } } }
I20260928 21:51:55.843930 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.842129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.3.0.194:43029)
0928 21:51:55.842166 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:55.842167 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:55.842175 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:51:55.842198 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:55.842199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:55.842211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:55.842263 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:51:55.842340 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.842777 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.843386 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:55.843562 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:55.843603 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:55.843698 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:51:55.843782 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:51:55.843834 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:55.843853 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 21:51:55.843868 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:51:55.845180 17530 raft_consensus.cc:1261] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Refusing update from remote peer 89225913bb0147d59cfe79b17ade11d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:51:55.845180 17805 raft_consensus.cc:1261] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c [term 1 FOLLOWER]: Refusing update from remote peer 89225913bb0147d59cfe79b17ade11d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:51:55.845443 17720 consensus_queue.cc:1035] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:51:55.845568 17864 consensus_queue.cc:1035] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:51:55.847525 17868 mvcc.cc:205] Tried to move back new op lower bound from 7334429965700165632 to 7334429965670477824. Current Snapshot: MvccSnapshot[applied={T|T < 7334429965700165632}]
I20260928 21:51:55.848130 17869 mvcc.cc:205] Tried to move back new op lower bound from 7334429965700165632 to 7334429965670477824. Current Snapshot: MvccSnapshot[applied={T|T < 7334429965700165632}]
I20260928 21:51:55.849370 17387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:51:55.849473 17387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:51:55.850725 17387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Sending DeleteTablet for 3 replicas of tablet aae8b894852f4df3b485ebb13a00e909
I20260928 21:51:55.851171 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 17427
I20260928 21:51:55.851200 17785 tablet_service.cc:1514] Processing DeleteTablet for tablet aae8b894852f4df3b485ebb13a00e909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:55 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:43542
I20260928 21:51:55.851368 17649 tablet_service.cc:1514] Processing DeleteTablet for tablet aae8b894852f4df3b485ebb13a00e909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:51:55 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:50878
I20260928 21:51:55.851445 17722 tablet_replica.cc:330] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0: stopping tablet replica
I20260928 21:51:55.851486 17722 raft_consensus.cc:2229] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:51:55.851553 17722 raft_consensus.cc:2258] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:51:55.851863 17722 ts_tablet_manager.cc:1905] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:51:55.853389 17722 ts_tablet_manager.cc:1918] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:51:55.853436 17722 log.cc:1196] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/aae8b894852f4df3b485ebb13a00e909
I20260928 21:51:55.853636 17722 ts_tablet_manager.cc:1939] T aae8b894852f4df3b485ebb13a00e909 P 89225913bb0147d59cfe79b17ade11d0: Deleting consensus metadata
I20260928 21:51:55.854091 17876 tablet_replica.cc:330] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: stopping tablet replica
I20260928 21:51:55.854079 17375 catalog_manager.cc:4841] TS 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029): tablet aae8b894852f4df3b485ebb13a00e909 (table test-table [id=4b99a566b41d4451ae46ba126c9e72a1]) successfully deleted
I20260928 21:51:55.854204 17876 raft_consensus.cc:2229] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:51:55.854312 17876 raft_consensus.cc:2258] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:51:55.854790 17876 ts_tablet_manager.cc:1905] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:51:55.856361 17876 ts_tablet_manager.cc:1918] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:51:55.856441 17876 log.cc:1196] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/aae8b894852f4df3b485ebb13a00e909
I20260928 21:51:55.856751 17876 ts_tablet_manager.cc:1939] T aae8b894852f4df3b485ebb13a00e909 P e2ea791b04894d549c45bb6e3732764c: Deleting consensus metadata
I20260928 21:51:55.858613 17375 catalog_manager.cc:4841] TS e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507): tablet aae8b894852f4df3b485ebb13a00e909 (table test-table [id=4b99a566b41d4451ae46ba126c9e72a1]) successfully deleted
W20260928 21:51:55.859138 17372 connection.cc:535] client connection to 127.3.0.193:34351 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:51:55.859246 17372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:51:55.859645 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 17578
I20260928 21:51:55.859769 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:55.859649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:34351 (local address 127.0.0.1:34672)
0928 21:51:55.859701 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:55.859734 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:34351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 21:51:55.860282 17372 catalog_manager.cc:4584] TS 20b3ae98089b444aaddde508a09cc75b (127.3.0.193:34351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aae8b894852f4df3b485ebb13a00e909: Network error: Client connection negotiation failed: client connection to 127.3.0.193:34351: connect: Connection refused (error 111)
I20260928 21:51:55.864823 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 17718
I20260928 21:51:55.868747 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 17355
I20260928 21:51:55.873194 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:36185
--webserver_interface=127.3.0.254
--webserver_port=37581
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:55.951493 17878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:55.951690 17878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:55.951711 17878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:55.952837 17878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:51:55.952878 17878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:55.952891 17878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:55.952901 17878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:51:55.952912 17878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:51:55.954332 17878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:36185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:55.954519 17878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:55.954681 17878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:55.959126 17886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:55.959131 17884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:55.959189 17883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:55.959270 17878 server_base.cc:1034] running on GCE node
I20260928 21:51:55.959667 17878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:55.959941 17878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:55.961082 17878 hybrid_clock.cc:648] HybridClock initialized: now 1790632315961063 us; error 33 us; skew 500 ppm
Sep 28 21:51:55 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:55.963357 17878 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:51:55.963778 17878 webserver.cc:460] Webserver started at http://127.3.0.254:37581/ using document root <none> and password file <none>
I20260928 21:51:55.963984 17878 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:55.964030 17878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:55.965325 17878 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:55.965931 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:55.966084 17878 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:55.966169 17878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "40d07be0c92a49b58eceb8bb2e31d6a5"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "4bdd7a90df6a3421d9bffaa6882c2973"
server_key_iv: "0cd5837a15ff0dad0ed3e2816798dadc"
server_key_version: "encryptionkey@0"
I20260928 21:51:55.966514 17878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:55.978293 17878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:55.980019 17878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:55.980170 17878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:55.984143 17878 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:36185
I20260928 21:51:55.984171 17945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:36185 every 8 connection(s)
I20260928 21:51:55.984570 17878 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:51:55.986047 17946 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:51:55.986632 17946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Bootstrap starting.
I20260928 21:51:55.987278 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 17878
I20260928 21:51:55.987519 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:34351
--local_ip_for_outbound_sockets=127.3.0.193
--tserver_master_addrs=127.3.0.254:36185
--webserver_port=39613
--webserver_interface=127.3.0.193
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:51:55.994596 17946 log.cc:824] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:55.996832 17946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:51:55.997089 17946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5: Bootstrap complete.
I20260928 21:51:55.999166 17946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:55.999406 17946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d07be0c92a49b58eceb8bb2e31d6a5, State: Initialized, Role: FOLLOWER
I20260928 21:51:55.999548 17946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:55.999605 17946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:51:55.999650 17946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:51:55.999694 17946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:51:56.000712 17946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:56.000813 17946 leader_election.cc:302] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40d07be0c92a49b58eceb8bb2e31d6a5; no voters:
I20260928 21:51:56.000962 17946 leader_election.cc:288] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 21:51:56.001008 17950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:51:56.001116 17950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [term 3 LEADER]: Becoming Leader. State: Replica: 40d07be0c92a49b58eceb8bb2e31d6a5, State: Running, Role: LEADER
I20260928 21:51:56.001204 17950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } }
I20260928 21:51:56.001233 17946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:51:56.001474 17951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } } }
I20260928 21:51:56.001560 17951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:56.001796 17952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d07be0c92a49b58eceb8bb2e31d6a5. Latest consensus state: current_term: 3 leader_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40d07be0c92a49b58eceb8bb2e31d6a5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 36185 } } }
I20260928 21:51:56.001892 17952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d07be0c92a49b58eceb8bb2e31d6a5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:51:56.001987 17954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:51:56.002784 17954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:51:56.002883 17954 catalog_manager.cc:1259] Loaded cluster ID: b8d3554a21504e99bf0fc91df0fc6b05
I20260928 21:51:56.002907 17954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 21:51:56.003151 17954 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa6d76f99ad google::LogMessage::Fail() at ??:0
@ 0x7fa6d76fc88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa6d76f94de google::LogMessage::Flush() at ??:0
@ 0x7fa6d76faa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa6df61a0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa6df6238b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa6d95c70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa6d95bc954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa6da4206db start_thread at ??:0
@ 0x7fa6d61f761f clone at ??:0
W20260928 21:51:56.074569 17948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:56.074777 17948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:56.074808 17948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:56.075939 17948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:56.075978 17948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:56.076013 17948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:51:56.077495 17948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:34351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=39613
--tserver_master_addrs=127.3.0.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:56.077723 17948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:56.077919 17948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:56.082556 17985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:56.082562 17982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:56.082818 17983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:56.082798 17948 server_base.cc:1034] running on GCE node
I20260928 21:51:56.082991 17948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:56.083220 17948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:56.084367 17948 hybrid_clock.cc:648] HybridClock initialized: now 1790632316084351 us; error 33 us; skew 500 ppm
Sep 28 21:51:56 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:56.086818 17948 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:51:56.087356 17948 webserver.cc:460] Webserver started at http://127.3.0.193:39613/ using document root <none> and password file <none>
I20260928 21:51:56.087601 17948 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:56.087664 17948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:56.089360 17948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:56.090087 17992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:56.090281 17948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:56.090361 17948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "20b3ae98089b444aaddde508a09cc75b"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "daf3149c7407a562b74937ccffd4df44"
server_key_iv: "666a3d800480231728c63cbc8c72d617"
server_key_version: "encryptionkey@0"
I20260928 21:51:56.090689 17948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:56.116623 17948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:56.118729 17948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:56.118845 17948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:56.119068 17948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:56.119524 17999 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:51:56.121405 17948 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:51:56.121448 17948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 21:51:56.121479 17948 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:51:56.122128 17948 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:51:56.122161 17948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 21:51:56.122248 17999 tablet_bootstrap.cc:492] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: Bootstrap starting.
I20260928 21:51:56.128561 17948 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:34351
I20260928 21:51:56.128710 18106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:34351 every 8 connection(s)
I20260928 21:51:56.128947 17948 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:51:56.132226 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 17948
I20260928 21:51:56.132504 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:43029
--local_ip_for_outbound_sockets=127.3.0.194
--tserver_master_addrs=127.3.0.254:36185
--webserver_port=43353
--webserver_interface=127.3.0.194
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:51:56.132776 17999 log.cc:824] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:51:56.135442 17999 tablet_bootstrap.cc:492] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:51:56.135735 17999 tablet_bootstrap.cc:492] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: Bootstrap complete.
I20260928 21:51:56.136858 17999 ts_tablet_manager.cc:1397] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260928 21:51:56.138212 17999 raft_consensus.cc:348] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:56.138533 17999 raft_consensus.cc:729] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b3ae98089b444aaddde508a09cc75b, State: Initialized, Role: FOLLOWER
I20260928 21:51:56.138686 17999 consensus_queue.cc:260] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:56.138947 17999 ts_tablet_manager.cc:1428] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260928 21:51:56.186290 18109 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:51:56.129594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:57417)
0928 21:51:56.129744 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:56.129786 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:56.130208 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:56.186083 (+ 55875us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
W20260928 21:51:56.186550 18107 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:36185 (0 consecutive failures): Network error: Failed to ping master at 127.3.0.254:36185: Client connection negotiation failed: client connection to 127.3.0.254:36185: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:51:56.186884 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:56.186770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:53355)
0928 21:51:56.186819 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:56.186859 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 21:51:56.187186 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:56.187094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:49863)
0928 21:51:56.187132 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:56.187156 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:51:56.187280 18107 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:51:56.214422 18111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:56.214617 18111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:56.214637 18111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:56.215772 18111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:56.215813 18111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:56.215842 18111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:51:56.217358 18111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:43029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=43353
--tserver_master_addrs=127.3.0.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:56.217582 18111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:56.217787 18111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:56.222525 18117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:56.222543 18111 server_base.cc:1034] running on GCE node
W20260928 21:51:56.222505 18118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:56.222525 18120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:56.222890 18111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:56.223109 18111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:56.224256 18111 hybrid_clock.cc:648] HybridClock initialized: now 1790632316224241 us; error 28 us; skew 500 ppm
Sep 28 21:51:56 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:56.226629 18111 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:51:56.227049 18111 webserver.cc:460] Webserver started at http://127.3.0.194:43353/ using document root <none> and password file <none>
I20260928 21:51:56.227259 18111 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:56.227306 18111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:56.228616 18111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:56.229117 18127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:56.229259 18111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:56.229338 18111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "89225913bb0147d59cfe79b17ade11d0"
format_stamp: "Formatted at 2026-09-28 21:51:54 on dist-test-slave-jrck"
server_key: "ef212f4e9f2b9802d9585a22b93903f9"
server_key_iv: "29b022a9e3adeab40705f30dffc79a1b"
server_key_version: "encryptionkey@0"
I20260928 21:51:56.229658 18111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:56.250231 18111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:56.252386 18111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:56.252543 18111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:56.252774 18111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:56.253059 18111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:56.253090 18111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:56.253119 18111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:56.253151 18111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:56.258620 18111 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:43029
I20260928 21:51:56.258683 18240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:43029 every 8 connection(s)
I20260928 21:51:56.258985 18111 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:51:56.259949 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:56.259582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:46479)
0928 21:51:56.259732 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:56.259809 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260928 21:51:56.260290 18241 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:36185 (0 consecutive failures): Network error: Failed to ping master at 127.3.0.254:36185: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
I20260928 21:51:56.260547 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:56.260444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:46717)
0928 21:51:56.260496 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:56.260525 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:51:56.260761 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:56.260697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:52411)
0928 21:51:56.260726 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:56.260743 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 21:51:56.260852 18241 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:51:56.268390 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 18111
I20260928 21:51:56.268667 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:36507
--local_ip_for_outbound_sockets=127.3.0.195
--tserver_master_addrs=127.3.0.254:36185
--webserver_port=42419
--webserver_interface=127.3.0.195
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:51:56.346776 18244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:51:56.346971 18244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:51:56.346992 18244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:51:56.348104 18244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:51:56.348143 18244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:51:56.348170 18244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:51:56.349623 18244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:36507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=42419
--tserver_master_addrs=127.3.0.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:51:56.349844 18244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:51:56.350035 18244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:51:56.354485 18252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:56.354518 18250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:51:56.354506 18249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:51:56.354588 18244 server_base.cc:1034] running on GCE node
I20260928 21:51:56.354872 18244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:51:56.355093 18244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:51:56.356226 18244 hybrid_clock.cc:648] HybridClock initialized: now 1790632316356212 us; error 31 us; skew 500 ppm
Sep 28 21:51:56 dist-test-slave-jrck krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:51:56.358624 18244 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:51:56.359066 18244 webserver.cc:460] Webserver started at http://127.3.0.195:42419/ using document root <none> and password file <none>
I20260928 21:51:56.359277 18244 fs_manager.cc:359] Metadata directory not provided
I20260928 21:51:56.359320 18244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:51:56.360570 18244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:51:56.361146 18259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:56.361299 18244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:51:56.361375 18244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "e2ea791b04894d549c45bb6e3732764c"
format_stamp: "Formatted at 2026-09-28 21:51:55 on dist-test-slave-jrck"
server_key: "0a9f78c725e0bfaed1323aeb83e70ad1"
server_key_iv: "753ae2e6a51f3db7c8c99e8621bd99d4"
server_key_version: "encryptionkey@0"
I20260928 21:51:56.361712 18244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:51:56.389816 18244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:51:56.392041 18244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:51:56.392158 18244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:51:56.392390 18244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:51:56.392731 18244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:51:56.392766 18244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:56.392796 18244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:51:56.392814 18244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:51:56.398205 18244 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:36507
I20260928 21:51:56.398247 18372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:36507 every 8 connection(s)
I20260928 21:51:56.398578 18244 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:51:56.399566 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:56.399205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:49125)
0928 21:51:56.399356 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:56.399426 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
W20260928 21:51:56.399955 18373 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:36185 (0 consecutive failures): Network error: Failed to ping master at 127.3.0.254:36185: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
I20260928 21:51:56.400213 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:56.400118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:57135)
0928 21:51:56.400158 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:56.400189 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 21:51:56.400446 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:56.400375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:39691)
0928 21:51:56.400408 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:56.400428 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 21:51:56.400542 18373 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:51:56.402958 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 18244
I20260928 21:51:57.187788 18377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:57.187581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:41923)
0928 21:51:57.187692 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:57.187725 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260928 21:51:57.261333 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:57.261144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:41889)
0928 21:51:57.261250 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:57.261284 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260928 21:51:57.401017 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:57.400834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:60989)
0928 21:51:57.400933 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:57.400969 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":54,"threads_started":1}
I20260928 21:51:57.928256 18380 raft_consensus.cc:482] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:51:57.928372 18380 raft_consensus.cc:504] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:51:57.928678 18380 leader_election.cc:288] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507), 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029)
Sep 28 21:51:57 dist-test-slave-jrck krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.194@KRBTEST.COM
Sep 28 21:51:57 dist-test-slave-jrck krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.195@KRBTEST.COM
I20260928 21:51:57.943656 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:57.928942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:58781 (local address 127.3.0.195:36507)
0928 21:51:57.929098 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:57.929111 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:57.929138 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:51:57.929610 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:57.929613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:57.929657 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:57.929764 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:57.930486 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:57.931331 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:57.932376 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:57.932853 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:57.933008 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:57.940704 (+ 7696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:57.940707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:57.940707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:57.940717 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:57.942284 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:57.942615 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:57.942618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:57.942619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:57.942730 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:57.942917 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:57.942917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:57.942917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:57.943145 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:57.943223 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:57.943417 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 21:51:57.943545 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":54,"threads_started":1}
I20260928 21:51:57.943670 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:57.929114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:38203 (local address 127.3.0.194:43029)
0928 21:51:57.929236 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:51:57.929246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:51:57.929262 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:51:57.929668 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:51:57.929670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:51:57.929707 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:51:57.929842 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:51:57.930487 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:57.931333 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:57.932365 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:51:57.932828 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:51:57.932922 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:57.940269 (+ 7347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:51:57.940274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:51:57.940274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:51:57.940287 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:51:57.942102 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:57.942451 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:57.942454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:57.942454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:57.942571 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:51:57.942806 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:51:57.942807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:51:57.942807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:51:57.943075 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:51:57.943133 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:51:57.943425 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 21:51:57.943562 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260928 21:51:57.943660 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:57.928823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:36507 (local address 127.3.0.193:58781)
0928 21:51:57.928932 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:57.928943 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:57.929009 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:57.929840 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:57.929843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:57.930223 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:57.930376 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:57.930381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:57.931424 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:57.931425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:57.932292 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:57.932295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:57.932948 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:57.932949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:57.933021 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:57.938759 (+ 5738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:57.938763 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:57.940600 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:57.942348 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:57.942349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:57.942349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:57.942532 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:57.942796 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:57.942797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:57.942797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:57.942859 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:57.943214 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:57.943215 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:57.943272 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 21:51:57.943356 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:51:57.943551 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":41,"thread_start_us":57,"threads_started":1}
I20260928 21:51:57.943993 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:57.928979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:43029 (local address 127.3.0.193:38203)
0928 21:51:57.929124 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:57.929132 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:51:57.929185 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:51:57.929851 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:51:57.929852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:51:57.930196 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:51:57.930376 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:57.930381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:57.931432 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:57.931433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:57.932297 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:51:57.932299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:51:57.932910 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:51:57.932911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:51:57.933028 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:51:57.938751 (+ 5723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:51:57.938760 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:51:57.940161 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:51:57.942176 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:57.942182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:57.942184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:57.942377 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:57.942638 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:51:57.942638 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:51:57.942639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:51:57.942751 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:51:57.943137 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:51:57.943144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:51:57.943274 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 21:51:57.943358 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:51:57.943620 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":219,"thread_start_us":41,"threads_started":1}
W20260928 21:51:57.944491 17996 leader_election.cc:341] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507): Not found: Tablet not found: aae8b894852f4df3b485ebb13a00e909
W20260928 21:51:57.944715 17996 leader_election.cc:341] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029): Not found: Tablet not found: aae8b894852f4df3b485ebb13a00e909
I20260928 21:51:57.944779 17996 leader_election.cc:302] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b3ae98089b444aaddde508a09cc75b; no voters: 89225913bb0147d59cfe79b17ade11d0, e2ea791b04894d549c45bb6e3732764c
I20260928 21:51:57.944898 18380 raft_consensus.cc:2726] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:51:58.188329 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:58.188206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:43677)
0928 21:51:58.188256 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:58.188290 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 21:51:58.261900 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:58.261693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:59501)
0928 21:51:58.261811 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:58.261852 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260928 21:51:58.401618 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:58.401410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:52737)
0928 21:51:58.401529 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:58.401573 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260928 21:51:59.188961 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:59.188723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:55037)
0928 21:51:59.188852 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:59.188898 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260928 21:51:59.262462 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:59.262284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:33371)
0928 21:51:59.262389 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:59.262422 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260928 21:51:59.402251 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:51:59.402019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:57201)
0928 21:51:59.402165 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:51:59.402207 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260928 21:52:00.135895 18390 raft_consensus.cc:482] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:52:00.136025 18390 raft_consensus.cc:504] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:52:00.136180 18390 leader_election.cc:288] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507), 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029)
W20260928 21:52:00.136622 17996 leader_election.cc:341] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507): Not found: Tablet not found: aae8b894852f4df3b485ebb13a00e909
W20260928 21:52:00.136698 17996 leader_election.cc:341] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029): Not found: Tablet not found: aae8b894852f4df3b485ebb13a00e909
I20260928 21:52:00.136729 17996 leader_election.cc:302] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b3ae98089b444aaddde508a09cc75b; no voters: 89225913bb0147d59cfe79b17ade11d0, e2ea791b04894d549c45bb6e3732764c
I20260928 21:52:00.136797 18390 raft_consensus.cc:2726] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:52:00.189596 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:00.189378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:57171)
0928 21:52:00.189503 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:00.189541 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260928 21:52:00.263018 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:00.262816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:54221)
0928 21:52:00.262927 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:00.262965 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260928 21:52:00.402910 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:00.402702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:54851)
0928 21:52:00.402817 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:00.402854 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260928 21:52:01.190474 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:01.190176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:42385)
0928 21:52:01.190369 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:01.190414 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
I20260928 21:52:01.263628 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:01.263403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:45223)
0928 21:52:01.263532 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:01.263570 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260928 21:52:01.403520 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:01.403324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:43089)
0928 21:52:01.403443 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:01.403476 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260928 21:52:02.191061 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:02.190870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:48527)
0928 21:52:02.190978 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:02.191012 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260928 21:52:02.264271 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:02.264063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:34637)
0928 21:52:02.264191 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:02.264229 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260928 21:52:02.404203 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:02.403980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:40911)
0928 21:52:02.404121 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:02.404160 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260928 21:52:02.732437 18400 raft_consensus.cc:482] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:52:02.732573 18400 raft_consensus.cc:504] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:52:02.732700 18400 leader_election.cc:288] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507), 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029)
W20260928 21:52:02.733103 17996 leader_election.cc:341] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029): Not found: Tablet not found: aae8b894852f4df3b485ebb13a00e909
W20260928 21:52:02.733173 17996 leader_election.cc:341] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507): Not found: Tablet not found: aae8b894852f4df3b485ebb13a00e909
I20260928 21:52:02.733187 17996 leader_election.cc:302] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b3ae98089b444aaddde508a09cc75b; no voters: 89225913bb0147d59cfe79b17ade11d0, e2ea791b04894d549c45bb6e3732764c
I20260928 21:52:02.733251 18400 raft_consensus.cc:2726] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:52:03.191658 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:03.191439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:34025)
0928 21:52:03.191574 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:03.191607 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260928 21:52:03.264822 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:03.264622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:51081)
0928 21:52:03.264742 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:03.264777 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260928 21:52:03.404879 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:03.404657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:59049)
0928 21:52:03.404790 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:03.404832 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260928 21:52:04.192354 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:04.192095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:43665)
0928 21:52:04.192248 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:04.192293 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260928 21:52:04.265545 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:04.265290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:60713)
0928 21:52:04.265455 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:04.265499 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260928 21:52:04.405604 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:04.405380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:55429)
0928 21:52:04.405514 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:04.405556 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260928 21:52:05.193012 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:05.192807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:41061)
0928 21:52:05.192923 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:05.192963 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260928 21:52:05.266155 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:05.265952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:48303)
0928 21:52:05.266078 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:05.266113 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260928 21:52:05.406317 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:05.406070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:57233)
0928 21:52:05.406224 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:05.406270 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
I20260928 21:52:06.193608 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:06.193416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:56579)
0928 21:52:06.193525 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:06.193560 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260928 21:52:06.266738 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:06.266549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:50393)
0928 21:52:06.266656 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:06.266693 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260928 21:52:06.406991 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:06.406771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:41763)
0928 21:52:06.406899 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:06.406941 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260928 21:52:07.173110 18413 raft_consensus.cc:482] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:52:07.173203 18413 raft_consensus.cc:504] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:52:07.173353 18413 leader_election.cc:288] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507), 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029)
W20260928 21:52:07.173725 17996 leader_election.cc:341] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507): Not found: Tablet not found: aae8b894852f4df3b485ebb13a00e909
W20260928 21:52:07.173799 17996 leader_election.cc:341] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029): Not found: Tablet not found: aae8b894852f4df3b485ebb13a00e909
I20260928 21:52:07.173815 17996 leader_election.cc:302] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b3ae98089b444aaddde508a09cc75b; no voters: 89225913bb0147d59cfe79b17ade11d0, e2ea791b04894d549c45bb6e3732764c
I20260928 21:52:07.173892 18413 raft_consensus.cc:2726] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:52:07.194262 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:07.194052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:60509)
0928 21:52:07.194188 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:07.194218 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":57,"threads_started":1}
I20260928 21:52:07.267334 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:07.267148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:52201)
0928 21:52:07.267259 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:07.267291 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260928 21:52:07.407876 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:07.407525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:53239)
0928 21:52:07.407736 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:07.407800 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
I20260928 21:52:08.194844 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:08.194632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:53925)
0928 21:52:08.194757 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:08.194795 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260928 21:52:08.267995 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:08.267744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:60973)
0928 21:52:08.267900 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:08.267951 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260928 21:52:08.408551 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:08.408299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:33495)
0928 21:52:08.408431 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:08.408495 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260928 21:52:08.732632 18420 raft_consensus.cc:482] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:52:08.732728 18420 raft_consensus.cc:504] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2ea791b04894d549c45bb6e3732764c" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 36507 } } peers { permanent_uuid: "89225913bb0147d59cfe79b17ade11d0" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 43029 } } peers { permanent_uuid: "20b3ae98089b444aaddde508a09cc75b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34351 } }
I20260928 21:52:08.732887 18420 leader_election.cc:288] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507), 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029)
W20260928 21:52:08.733234 17996 leader_election.cc:341] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2ea791b04894d549c45bb6e3732764c (127.3.0.195:36507): Not found: Tablet not found: aae8b894852f4df3b485ebb13a00e909
W20260928 21:52:08.733306 17996 leader_election.cc:341] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 89225913bb0147d59cfe79b17ade11d0 (127.3.0.194:43029): Not found: Tablet not found: aae8b894852f4df3b485ebb13a00e909
I20260928 21:52:08.733352 17996 leader_election.cc:302] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20b3ae98089b444aaddde508a09cc75b; no voters: 89225913bb0147d59cfe79b17ade11d0, e2ea791b04894d549c45bb6e3732764c
I20260928 21:52:08.733435 18420 raft_consensus.cc:2726] T aae8b894852f4df3b485ebb13a00e909 P 20b3ae98089b444aaddde508a09cc75b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:52:09.195420 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:09.195210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:53321)
0928 21:52:09.195327 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:09.195370 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260928 21:52:09.268720 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:09.268414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:34303)
0928 21:52:09.268575 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:09.268618 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260928 21:52:09.409205 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:09.408971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:34707)
0928 21:52:09.409118 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:09.409160 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260928 21:52:10.195989 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:10.195794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:43383)
0928 21:52:10.195901 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:10.195936 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260928 21:52:10.269331 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:10.269133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:48925)
0928 21:52:10.269253 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:10.269286 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260928 21:52:10.409864 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:10.409614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:43627)
0928 21:52:10.409779 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:10.409817 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260928 21:52:11.196695 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:11.196483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.193:58515)
0928 21:52:11.196601 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:11.196637 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260928 21:52:11.269979 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:11.269770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.194:56861)
0928 21:52:11.269897 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:11.269935 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260928 21:52:11.410573 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:11.410366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:36185 (local address 127.3.0.195:38695)
0928 21:52:11.410488 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:11.410528 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:36185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260928 21:52:11.413120 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 17948
I20260928 21:52:11.417182 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 18111
I20260928 21:52:11.420960 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 18244
2026-09-28T21:52:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17945 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:52:11 dist-test-slave-jrck krb5kdc[18432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:52:11 dist-test-slave-jrck krb5kdc[18432](info): set up 2 sockets
Sep 28 21:52:11 dist-test-slave-jrck krb5kdc[18432](info): commencing operation
krb5kdc: starting...
W20260928 21:52:12.376124 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:52:12 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:52:12.482126 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38227
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:52:12.558873 18445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:12.559069 18445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:12.559089 18445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:12.560204 18445 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:52:12.560242 18445 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:52:12.560254 18445 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:52:12.560266 18445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:52:12.560289 18445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:12.560315 18445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:12.560338 18445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:52:12.560357 18445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:52:12.561806 18445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:38227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:12.562038 18445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:12.562237 18445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:52:12.566915 18445 server_base.cc:1034] running on GCE node
W20260928 21:52:12.566903 18453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:12.566927 18451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:12.566895 18450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:12.567332 18445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:12.567611 18445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:12.568740 18445 hybrid_clock.cc:648] HybridClock initialized: now 1790632332568724 us; error 33 us; skew 500 ppm
Sep 28 21:52:12 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:12.570963 18445 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:52:12.571334 18445 webserver.cc:460] Webserver started at http://127.3.0.254:33691/ using document root <none> and password file <none>
I20260928 21:52:12.571496 18445 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:12.571532 18445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:12.571645 18445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:52:12.572649 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "a347157b6f7b4abe8f144601793d40df"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "520d331bb7184312f3186b5cd45f35f0"
server_key_iv: "3a2a503955b3b5e7cfe8d13347ae3753"
server_key_version: "encryptionkey@0"
I20260928 21:52:12.572929 18445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "a347157b6f7b4abe8f144601793d40df"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "520d331bb7184312f3186b5cd45f35f0"
server_key_iv: "3a2a503955b3b5e7cfe8d13347ae3753"
server_key_version: "encryptionkey@0"
I20260928 21:52:12.574040 18445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:12.574652 18460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:12.574795 18445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:52:12.574873 18445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "a347157b6f7b4abe8f144601793d40df"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "520d331bb7184312f3186b5cd45f35f0"
server_key_iv: "3a2a503955b3b5e7cfe8d13347ae3753"
server_key_version: "encryptionkey@0"
I20260928 21:52:12.574930 18445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:12.582100 18445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:12.584024 18445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:12.584147 18445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:12.587666 18445 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:38227
I20260928 21:52:12.587700 18512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:38227 every 8 connection(s)
I20260928 21:52:12.587977 18445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:52:12.588768 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:52:12.590868 18513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Bootstrap starting.
I20260928 21:52:12.591374 18513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Neither blocks nor log segments found. Creating new log.
I20260928 21:52:12.591594 18513 log.cc:824] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Log is configured to *not* fsync() on all Append() calls
I20260928 21:52:12.592170 18513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: No bootstrap required, opened a new log
I20260928 21:52:12.593672 18513 raft_consensus.cc:348] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:12.593801 18513 raft_consensus.cc:374] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:52:12.593822 18513 raft_consensus.cc:729] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a347157b6f7b4abe8f144601793d40df, State: Initialized, Role: FOLLOWER
I20260928 21:52:12.593923 18513 consensus_queue.cc:260] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:12.594003 18513 raft_consensus.cc:388] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:52:12.594030 18513 raft_consensus.cc:482] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:52:12.594061 18513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:52:12.594499 18513 raft_consensus.cc:504] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:12.594578 18513 leader_election.cc:302] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a347157b6f7b4abe8f144601793d40df; no voters:
I20260928 21:52:12.594698 18513 leader_election.cc:288] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:52:12.594753 18516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:52:12.594906 18513 sys_catalog.cc:564] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:52:12.594918 18516 raft_consensus.cc:686] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 1 LEADER]: Becoming Leader. State: Replica: a347157b6f7b4abe8f144601793d40df, State: Running, Role: LEADER
I20260928 21:52:12.595075 18516 consensus_queue.cc:237] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:12.595415 18517 sys_catalog.cc:455] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a347157b6f7b4abe8f144601793d40df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } } }
I20260928 21:52:12.595521 18517 sys_catalog.cc:458] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: This master's current role is: LEADER
I20260928 21:52:12.595402 18518 sys_catalog.cc:455] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: SysCatalogTable state changed. Reason: New leader a347157b6f7b4abe8f144601793d40df. Latest consensus state: current_term: 1 leader_uuid: "a347157b6f7b4abe8f144601793d40df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } } }
I20260928 21:52:12.595745 18518 sys_catalog.cc:458] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: This master's current role is: LEADER
I20260928 21:52:12.595780 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 18445
I20260928 21:52:12.595863 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:52:12.596062 3075 external_mini_cluster.cc:1428] Setting key 782719319d326938d9324176fe751fda
I20260928 21:52:12.596139 18524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:52:12.596644 18524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:52:12.599279 18524 catalog_manager.cc:1347] Generated new cluster ID: de27b1c2e9b24b6894d387e460a99b39
I20260928 21:52:12.599334 18524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:52:12 dist-test-slave-jrck krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:52:12.605804 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:12.596512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.3.0.254:38227)
0928 21:52:12.596682 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:12.596692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:12.596727 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:52:12.596898 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:12.596906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:12.596966 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:12.597102 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:12.597820 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:12.599139 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:12.599903 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:12.600324 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:12.600416 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:12.602486 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:12.602490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:12.602490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:12.602501 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:12.604422 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:12.604769 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:12.604772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:12.604772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:12.604853 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:12.605022 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:12.605023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:12.605024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:12.605244 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:12.605376 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:12.605481 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 21:52:12.605620 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260928 21:52:12.614744 18524 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:52:12.615228 18524 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:52:12.622148 18524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Generated new TSK 0
I20260928 21:52:12.622301 18524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:52:12.652653 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38227
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:52:12.751159 18539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:12.751360 18539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:12.751382 18539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:12.752525 18539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:12.752562 18539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:12.752586 18539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:52:12.754011 18539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:12.754212 18539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:12.754398 18539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:52:12.759020 18545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:12.759027 18544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:12.759027 18547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:12.759140 18539 server_base.cc:1034] running on GCE node
I20260928 21:52:12.759524 18539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:12.759758 18539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:12.760936 18539 hybrid_clock.cc:648] HybridClock initialized: now 1790632332760920 us; error 31 us; skew 500 ppm
Sep 28 21:52:12 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:12.763549 18539 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:52:12.763957 18539 webserver.cc:460] Webserver started at http://127.3.0.193:35741/ using document root <none> and password file <none>
I20260928 21:52:12.764169 18539 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:12.764215 18539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:12.764338 18539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:52:12.765404 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "0b755691f36c4e689d712cb193612fcf"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "0c8e166ff84aef515d1899a6b606ac05"
server_key_iv: "759efaaffb76fb7aba219f3a69f18bbf"
server_key_version: "encryptionkey@0"
I20260928 21:52:12.765734 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "0b755691f36c4e689d712cb193612fcf"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "0c8e166ff84aef515d1899a6b606ac05"
server_key_iv: "759efaaffb76fb7aba219f3a69f18bbf"
server_key_version: "encryptionkey@0"
I20260928 21:52:12.766922 18539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:52:12.767617 18554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:12.767812 18539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:52:12.767899 18539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "0b755691f36c4e689d712cb193612fcf"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "0c8e166ff84aef515d1899a6b606ac05"
server_key_iv: "759efaaffb76fb7aba219f3a69f18bbf"
server_key_version: "encryptionkey@0"
I20260928 21:52:12.767966 18539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:12.779397 18539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:12.781286 18539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:12.781399 18539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:12.781598 18539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:52:12.781862 18539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:52:12.781893 18539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:12.781913 18539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:52:12.781936 18539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:12.787271 18539 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:35837
I20260928 21:52:12.787326 18667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:35837 every 8 connection(s)
I20260928 21:52:12.787603 18539 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:52:12 dist-test-slave-jrck krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:52:12.795877 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:12.788224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:58143 (local address 127.3.0.254:38227)
0928 21:52:12.788267 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:12.788269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:12.788449 (+ 180us) server_negotiation.cc:407] Connection header received
0928 21:52:12.788975 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:12.788976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:12.788992 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:12.789050 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:12.789945 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:12.790449 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:12.791420 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:12.791634 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:12.791688 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:12.793665 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:12.793666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:12.793667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:12.793674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:12.794745 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:12.795079 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:12.795080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:12.795080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:12.795124 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:12.795342 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:12.795343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:12.795343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:12.795480 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:12.795530 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:12.795748 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 21:52:12.795814 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:52:12.796012 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:12.788195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:58143)
0928 21:52:12.788334 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:12.788371 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:12.788874 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:12.789112 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:12.789116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:12.789664 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:52:12.789847 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:12.789852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:12.790532 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:12.790533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:12.791334 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:12.791336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:12.791703 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:12.791704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:12.791830 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:12.792225 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:52:12.792232 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:52:12.793583 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:52:12.794808 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:12.794813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:12.794815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:12.795024 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:12.795168 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:12.795169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:12.795169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:12.795293 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:12.795530 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:52:12.795535 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:52:12.795648 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 21:52:12.795728 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:52:12.795841 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260928 21:52:12.796512 18668 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38227
I20260928 21:52:12.796593 18668 heartbeater.cc:464] Registering TS with master...
I20260928 21:52:12.796746 18668 heartbeater.cc:511] Master 127.3.0.254:38227 requested a full tablet report, sending...
I20260928 21:52:12.797217 18477 ts_manager.cc:194] Registered new tserver with Master: 0b755691f36c4e689d712cb193612fcf (127.3.0.193:35837)
I20260928 21:52:12.797767 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 18539
I20260928 21:52:12.797858 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:52:12.797842 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:58143
I20260928 21:52:12.798044 3075 external_mini_cluster.cc:1428] Setting key 26a43c45d260c57b7732b38c9c2c862f
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:52:12.818459 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38227
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:52:12.896646 18673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:12.896852 18673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:12.896873 18673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:12.897987 18673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:12.898022 18673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:12.898046 18673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:52:12.899405 18673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:12.899600 18673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:12.899770 18673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:52:12.904322 18681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:12.904323 18679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:12.904350 18678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:12.904337 18673 server_base.cc:1034] running on GCE node
I20260928 21:52:12.904721 18673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:12.904973 18673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:12.906142 18673 hybrid_clock.cc:648] HybridClock initialized: now 1790632332906132 us; error 34 us; skew 500 ppm
Sep 28 21:52:12 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:12.908440 18673 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:52:12.908914 18673 webserver.cc:460] Webserver started at http://127.3.0.194:39567/ using document root <none> and password file <none>
I20260928 21:52:12.909112 18673 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:12.909158 18673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:12.909277 18673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:52:12.910256 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "ee397f845f8047979795c00290b51aee"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "3dc548adc42e580fbcc139c2044c0a65"
server_key_iv: "2a5903cccb4f36af0b329930cae36f24"
server_key_version: "encryptionkey@0"
I20260928 21:52:12.910575 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "ee397f845f8047979795c00290b51aee"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "3dc548adc42e580fbcc139c2044c0a65"
server_key_iv: "2a5903cccb4f36af0b329930cae36f24"
server_key_version: "encryptionkey@0"
I20260928 21:52:12.911847 18673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 21:52:12.912509 18688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:12.912659 18673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:12.912738 18673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "ee397f845f8047979795c00290b51aee"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "3dc548adc42e580fbcc139c2044c0a65"
server_key_iv: "2a5903cccb4f36af0b329930cae36f24"
server_key_version: "encryptionkey@0"
I20260928 21:52:12.912804 18673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:12.925165 18673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:12.926725 18673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:12.926853 18673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:12.927045 18673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:52:12.927307 18673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:52:12.927338 18673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:12.927364 18673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:52:12.927380 18673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:12.932792 18673 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:37769
I20260928 21:52:12.932850 18801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:37769 every 8 connection(s)
I20260928 21:52:12.933126 18673 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:52:12 dist-test-slave-jrck krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:52:12.941105 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:12.933740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:58877 (local address 127.3.0.254:38227)
0928 21:52:12.933798 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:12.933800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:12.933955 (+ 155us) server_negotiation.cc:407] Connection header received
0928 21:52:12.934420 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:12.934421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:12.934430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:12.934479 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:12.935320 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:12.935800 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:12.936816 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:12.937022 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:12.937081 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:12.938896 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:12.938897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:12.938897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:12.938904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:12.940008 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:12.940355 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:12.940356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:12.940356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:12.940400 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:12.940633 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:12.940634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:12.940634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:12.940735 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:12.940788 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:12.940984 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 21:52:12.941044 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:52:12.941248 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:12.933693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:58877)
0928 21:52:12.933847 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:12.933886 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:12.934347 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:12.934546 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:12.934549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:12.935076 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:52:12.935257 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:12.935262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:12.935862 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:12.935863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:12.936725 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:12.936728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:12.937087 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:12.937088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:12.937185 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:12.937590 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:52:12.937597 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:52:12.938816 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:52:12.940087 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:12.940094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:12.940096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:12.940300 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:12.940444 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:12.940445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:12.940445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:12.940577 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:12.940783 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:52:12.940786 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:52:12.940892 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 21:52:12.940965 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:52:12.941078 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260928 21:52:12.941655 18802 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38227
I20260928 21:52:12.941736 18802 heartbeater.cc:464] Registering TS with master...
I20260928 21:52:12.941867 18802 heartbeater.cc:511] Master 127.3.0.254:38227 requested a full tablet report, sending...
I20260928 21:52:12.942210 18477 ts_manager.cc:194] Registered new tserver with Master: ee397f845f8047979795c00290b51aee (127.3.0.194:37769)
I20260928 21:52:12.942552 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:58877
I20260928 21:52:12.942618 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 18673
I20260928 21:52:12.942716 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:52:12.942931 3075 external_mini_cluster.cc:1428] Setting key 17ef6287ee04722596eb13e82e66204f
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:52:12.963306 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38227
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:52:13.043152 18807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:13.043354 18807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:13.043375 18807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:13.044494 18807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:13.044528 18807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:13.044552 18807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:52:13.045943 18807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:13.046128 18807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:13.046305 18807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:52:13.050868 18812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:13.050881 18813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:13.050881 18815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:13.051174 18807 server_base.cc:1034] running on GCE node
I20260928 21:52:13.051365 18807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:13.051579 18807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:13.052713 18807 hybrid_clock.cc:648] HybridClock initialized: now 1790632333052699 us; error 23 us; skew 500 ppm
Sep 28 21:52:13 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632333, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:13.055035 18807 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:52:13.055402 18807 webserver.cc:460] Webserver started at http://127.3.0.195:40587/ using document root <none> and password file <none>
I20260928 21:52:13.055572 18807 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:13.055610 18807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:13.055698 18807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:52:13.056720 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "3c8f1afe27ff436e99f9cda9b87d2e40"
format_stamp: "Formatted at 2026-09-28 21:52:13 on dist-test-slave-jrck"
server_key: "7c54591c844e036061c2fc97476ac422"
server_key_iv: "1eb152d7c4ff077ba5807c5a6ce91919"
server_key_version: "encryptionkey@0"
I20260928 21:52:13.057001 18807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "3c8f1afe27ff436e99f9cda9b87d2e40"
format_stamp: "Formatted at 2026-09-28 21:52:13 on dist-test-slave-jrck"
server_key: "7c54591c844e036061c2fc97476ac422"
server_key_iv: "1eb152d7c4ff077ba5807c5a6ce91919"
server_key_version: "encryptionkey@0"
I20260928 21:52:13.058144 18807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:13.058768 18822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:13.058892 18807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:52:13.058959 18807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "3c8f1afe27ff436e99f9cda9b87d2e40"
format_stamp: "Formatted at 2026-09-28 21:52:13 on dist-test-slave-jrck"
server_key: "7c54591c844e036061c2fc97476ac422"
server_key_iv: "1eb152d7c4ff077ba5807c5a6ce91919"
server_key_version: "encryptionkey@0"
I20260928 21:52:13.059007 18807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:13.070211 18807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:13.071945 18807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:13.072060 18807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:13.072230 18807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:52:13.072494 18807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:52:13.072531 18807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:13.072552 18807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:52:13.072562 18807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:13.077936 18807 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:37381
I20260928 21:52:13.078032 18935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:37381 every 8 connection(s)
I20260928 21:52:13.078292 18807 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:52:13 dist-test-slave-jrck krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632333, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:52:13.086333 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.078918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:58205 (local address 127.3.0.254:38227)
0928 21:52:13.078976 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:13.078979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:13.079189 (+ 210us) server_negotiation.cc:407] Connection header received
0928 21:52:13.079651 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:13.079652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:13.079663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:13.079712 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:13.080613 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.081135 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.082080 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.082260 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.082306 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.084144 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:13.084145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:13.084146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:13.084152 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:13.085304 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:13.085644 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:13.085645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:13.085645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:13.085689 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:13.085859 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:13.085859 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:13.085860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:13.085951 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:13.085997 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:13.086202 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 21:52:13.086266 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:52:13.086486 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.078905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:58205)
0928 21:52:13.079076 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:13.079111 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:13.079567 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:13.079767 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:13.079770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:13.080310 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:52:13.080543 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.080549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.081212 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.081213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.082005 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.082007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.082313 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.082314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.082404 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.082799 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:52:13.082806 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:52:13.084067 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:52:13.085366 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:13.085372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:13.085374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:13.085588 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:13.085732 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:13.085733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:13.085733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:13.085820 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:13.085993 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:52:13.085995 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:52:13.086109 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 21:52:13.086182 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:52:13.086298 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260928 21:52:13.086905 18936 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38227
I20260928 21:52:13.086997 18936 heartbeater.cc:464] Registering TS with master...
I20260928 21:52:13.087167 18936 heartbeater.cc:511] Master 127.3.0.254:38227 requested a full tablet report, sending...
I20260928 21:52:13.087533 18477 ts_manager.cc:194] Registered new tserver with Master: 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381)
I20260928 21:52:13.087779 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 18807
I20260928 21:52:13.087867 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:52:13.087910 18477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:58205
I20260928 21:52:13.088059 3075 external_mini_cluster.cc:1428] Setting key 567e7336ae64294a4be8d6bd6d40ee08
I20260928 21:52:13.088848 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:52:13.095202 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.090074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.3.0.254:38227)
0928 21:52:13.090128 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:13.090129 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:13.090192 (+ 63us) server_negotiation.cc:407] Connection header received
0928 21:52:13.090257 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:13.090258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:13.090267 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:13.090307 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:13.090833 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.091297 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.092002 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.092176 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.092228 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.093247 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:13.093248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:13.093248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:13.093255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:13.094199 (+ 944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:13.094478 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:13.094479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:13.094479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:13.094530 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:13.094682 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:13.094682 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:13.094683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:13.094816 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:13.094902 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:13.095051 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:52:13.095121 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:52:13.096549 18477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:52:13.098073 18477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:52:13.103797 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.099859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:37769 (local address 127.0.0.1:55266)
0928 21:52:13.100118 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:13.100128 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:13.100176 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:13.100862 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:13.100863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:13.100865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:52:13.100901 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.100902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.102068 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.102069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.102916 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.102919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.103527 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.103528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.103604 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.103616 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 21:52:13.103678 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:52:13.103728 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":45,"threads_started":1}
I20260928 21:52:13.103809 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.099956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.3.0.194:37769)
0928 21:52:13.100642 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:13.100647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:13.100656 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:52:13.100693 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:13.100695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:13.100709 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:13.100784 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:52:13.100988 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.101987 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.102987 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.103457 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.103518 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.103576 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:13.103696 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 21:52:13.103730 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":83,"threads_started":1}
I20260928 21:52:13.104152 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.099849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:35837 (local address 127.0.0.1:58166)
0928 21:52:13.099998 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:13.100039 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:13.100098 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:13.100527 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:13.100531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:13.100533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:52:13.100610 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.100612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.101675 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.101676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.102948 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.102950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.103887 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.103888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.103958 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.103962 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:52:13.104015 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:52:13.104070 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":68,"thread_start_us":52,"threads_started":1}
I20260928 21:52:13.104167 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.099975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.3.0.193:35837)
0928 21:52:13.100334 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:13.100338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:13.100347 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:52:13.100383 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:13.100385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:13.100399 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:13.100519 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:52:13.100750 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.101581 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.103034 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.103736 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.103799 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.103851 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:13.104045 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 21:52:13.104087 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260928 21:52:13.104152 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.100029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:37381 (local address 127.0.0.1:47326)
0928 21:52:13.100181 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:13.100189 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:13.100225 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:13.100535 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:13.100535 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:13.100536 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:52:13.100610 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.100612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.101867 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.101868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.102939 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.102941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.103887 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.103888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.103963 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.103966 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:52:13.104006 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 21:52:13.104066 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":5,"thread_start_us":41,"threads_started":1}
I20260928 21:52:13.104300 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.100181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.3.0.195:37381)
0928 21:52:13.100333 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:13.100337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:13.100347 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:52:13.100382 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:13.100384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:13.100399 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:13.100520 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:52:13.100696 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.101770 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.103041 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.103733 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.103799 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.103851 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:13.104188 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 21:52:13.104227 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260928 21:52:13.104803 18736 tablet_service.cc:1467] Processing CreateTablet for tablet 532611d7ad144308ab745f2bc2c521bb (DEFAULT_TABLE table=test-table [id=f9fb54df387f471297a0d37e9f72fc08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:52:13.105154 18736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 532611d7ad144308ab745f2bc2c521bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:52:13.105221 18870 tablet_service.cc:1467] Processing CreateTablet for tablet 532611d7ad144308ab745f2bc2c521bb (DEFAULT_TABLE table=test-table [id=f9fb54df387f471297a0d37e9f72fc08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:52:13.105515 18870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 532611d7ad144308ab745f2bc2c521bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:52:13.107656 18950 tablet_bootstrap.cc:492] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: Bootstrap starting.
I20260928 21:52:13.108009 18951 tablet_bootstrap.cc:492] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: Bootstrap starting.
I20260928 21:52:13.107971 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 532611d7ad144308ab745f2bc2c521bb (DEFAULT_TABLE table=test-table [id=f9fb54df387f471297a0d37e9f72fc08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:52:13.108275 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 532611d7ad144308ab745f2bc2c521bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:52:13.108327 18950 tablet_bootstrap.cc:654] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: Neither blocks nor log segments found. Creating new log.
I20260928 21:52:13.108660 18951 tablet_bootstrap.cc:654] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: Neither blocks nor log segments found. Creating new log.
I20260928 21:52:13.108842 18950 log.cc:824] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: Log is configured to *not* fsync() on all Append() calls
I20260928 21:52:13.109218 18951 log.cc:824] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: Log is configured to *not* fsync() on all Append() calls
I20260928 21:52:13.109524 18950 tablet_bootstrap.cc:492] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: No bootstrap required, opened a new log
I20260928 21:52:13.109593 18950 ts_tablet_manager.cc:1397] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:52:13.109958 18951 tablet_bootstrap.cc:492] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: No bootstrap required, opened a new log
I20260928 21:52:13.110038 18951 ts_tablet_manager.cc:1397] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:52:13.110960 18950 raft_consensus.cc:348] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:13.111204 18950 raft_consensus.cc:374] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:52:13.111296 18950 raft_consensus.cc:729] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee397f845f8047979795c00290b51aee, State: Initialized, Role: FOLLOWER
I20260928 21:52:13.111438 18950 consensus_queue.cc:260] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:13.111692 18950 ts_tablet_manager.cc:1428] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:52:13.111722 18954 tablet_bootstrap.cc:492] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Bootstrap starting.
I20260928 21:52:13.111804 18802 heartbeater.cc:503] Master 127.3.0.254:38227 was elected leader, sending a full tablet report...
I20260928 21:52:13.111879 18951 raft_consensus.cc:348] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:13.112056 18951 raft_consensus.cc:374] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:52:13.112136 18951 raft_consensus.cc:729] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8f1afe27ff436e99f9cda9b87d2e40, State: Initialized, Role: FOLLOWER
I20260928 21:52:13.112253 18951 consensus_queue.cc:260] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:13.112435 18954 tablet_bootstrap.cc:654] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Neither blocks nor log segments found. Creating new log.
I20260928 21:52:13.113052 18954 log.cc:824] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Log is configured to *not* fsync() on all Append() calls
I20260928 21:52:13.113511 18936 heartbeater.cc:503] Master 127.3.0.254:38227 was elected leader, sending a full tablet report...
I20260928 21:52:13.113706 18951 ts_tablet_manager.cc:1428] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 21:52:13.113746 18954 tablet_bootstrap.cc:492] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: No bootstrap required, opened a new log
I20260928 21:52:13.113828 18954 ts_tablet_manager.cc:1397] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:52:13.115386 18954 raft_consensus.cc:348] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:13.115543 18954 raft_consensus.cc:374] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:52:13.115566 18954 raft_consensus.cc:729] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b755691f36c4e689d712cb193612fcf, State: Initialized, Role: FOLLOWER
I20260928 21:52:13.115643 18954 consensus_queue.cc:260] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:13.115821 18954 ts_tablet_manager.cc:1428] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:52:13.115883 18668 heartbeater.cc:503] Master 127.3.0.254:38227 was elected leader, sending a full tablet report...
W20260928 21:52:13.183662 18803 tablet.cc:2369] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:52:13.288239 18669 tablet.cc:2369] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:52:13.328778 18937 tablet.cc:2369] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:52:13.367659 18955 raft_consensus.cc:482] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:52:13.367779 18955 raft_consensus.cc:504] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:13.368142 18955 leader_election.cc:288] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b755691f36c4e689d712cb193612fcf (127.3.0.193:35837), 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381)
I20260928 21:52:13.371018 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.368318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:37381 (local address 127.3.0.194:53209)
0928 21:52:13.368362 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:13.368373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:13.368415 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:13.368591 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:13.368592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:13.368594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:52:13.368660 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.368661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.369363 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.369364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.370203 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.370206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.370759 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.370760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.370840 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.370845 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:52:13.370899 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:52:13.370951 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 21:52:13.371019 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.368370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:53209 (local address 127.3.0.195:37381)
0928 21:52:13.368420 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:13.368422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:13.368434 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:52:13.368487 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:13.368488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:13.368500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:13.368546 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:52:13.368729 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.369279 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.370276 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.370699 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.370747 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.370756 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:13.370917 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 21:52:13.370950 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:52:13.371177 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.368318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:35837 (local address 127.3.0.194:60263)
0928 21:52:13.368661 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:13.368668 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:13.368719 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:13.368831 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:13.368832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:13.368834 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:52:13.368901 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.368903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.369591 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.369592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.370378 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:13.370380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.370933 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.370934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:13.371008 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.371012 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:52:13.371068 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:52:13.371106 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":163,"thread_start_us":112,"threads_started":1}
I20260928 21:52:13.371196 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.368367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:60263 (local address 127.3.0.193:35837)
0928 21:52:13.368426 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:13.368427 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:13.368726 (+ 299us) server_negotiation.cc:407] Connection header received
0928 21:52:13.368761 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:13.368762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:13.368775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:13.368820 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:52:13.368992 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.369513 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.370463 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.370862 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.370921 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.370930 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:13.371079 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:52:13.371117 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:52:13.371363 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "532611d7ad144308ab745f2bc2c521bb" candidate_uuid: "ee397f845f8047979795c00290b51aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" is_pre_election: true
I20260928 21:52:13.371532 18890 raft_consensus.cc:2454] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee397f845f8047979795c00290b51aee in term 0.
I20260928 21:52:13.371476 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "532611d7ad144308ab745f2bc2c521bb" candidate_uuid: "ee397f845f8047979795c00290b51aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b755691f36c4e689d712cb193612fcf" is_pre_election: true
I20260928 21:52:13.371626 18622 raft_consensus.cc:2454] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee397f845f8047979795c00290b51aee in term 0.
I20260928 21:52:13.371737 18689 leader_election.cc:302] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8f1afe27ff436e99f9cda9b87d2e40, ee397f845f8047979795c00290b51aee; no voters:
I20260928 21:52:13.371852 18955 raft_consensus.cc:2781] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:52:13.371915 18955 raft_consensus.cc:482] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:52:13.371937 18955 raft_consensus.cc:3037] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:52:13.372632 18955 raft_consensus.cc:504] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:13.372742 18955 leader_election.cc:288] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [CANDIDATE]: Term 1 election: Requested vote from peers 0b755691f36c4e689d712cb193612fcf (127.3.0.193:35837), 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381)
I20260928 21:52:13.372920 18622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "532611d7ad144308ab745f2bc2c521bb" candidate_uuid: "ee397f845f8047979795c00290b51aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b755691f36c4e689d712cb193612fcf"
I20260928 21:52:13.372907 18890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "532611d7ad144308ab745f2bc2c521bb" candidate_uuid: "ee397f845f8047979795c00290b51aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40"
I20260928 21:52:13.372994 18890 raft_consensus.cc:3037] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:52:13.372994 18622 raft_consensus.cc:3037] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:52:13.373816 18622 raft_consensus.cc:2454] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee397f845f8047979795c00290b51aee in term 1.
I20260928 21:52:13.373816 18890 raft_consensus.cc:2454] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee397f845f8047979795c00290b51aee in term 1.
I20260928 21:52:13.373958 18689 leader_election.cc:302] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c8f1afe27ff436e99f9cda9b87d2e40, ee397f845f8047979795c00290b51aee; no voters:
I20260928 21:52:13.374061 18955 raft_consensus.cc:2781] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:52:13.374187 18955 raft_consensus.cc:686] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 1 LEADER]: Becoming Leader. State: Replica: ee397f845f8047979795c00290b51aee, State: Running, Role: LEADER
I20260928 21:52:13.374279 18955 consensus_queue.cc:237] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:13.375044 18477 catalog_manager.cc:5495] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee reported cstate change: term changed from 0 to 1, leader changed from <none> to ee397f845f8047979795c00290b51aee (127.3.0.194). New cstate: current_term: 1 leader_uuid: "ee397f845f8047979795c00290b51aee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } health_report { overall_health: UNKNOWN } } }
I20260928 21:52:13.383231 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.381276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.3.0.194:37769)
0928 21:52:13.381312 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:13.381314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:13.381323 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:52:13.381385 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:13.381386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:13.381401 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:13.381454 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:52:13.381545 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.381963 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.382652 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:13.382831 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:13.382875 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:13.382974 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:52:13.383072 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:52:13.383130 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:13.383146 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 21:52:13.383165 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:52:13.384704 18622 raft_consensus.cc:1261] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Refusing update from remote peer ee397f845f8047979795c00290b51aee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:52:13.384711 18890 raft_consensus.cc:1261] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 1 FOLLOWER]: Refusing update from remote peer ee397f845f8047979795c00290b51aee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:52:13.384970 18955 consensus_queue.cc:1035] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:52:13.385104 18960 consensus_queue.cc:1035] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:52:13.387106 18963 mvcc.cc:205] Tried to move back new op lower bound from 7334430037541621760 to 7334430037501775872. Current Snapshot: MvccSnapshot[applied={T|T < 7334430037541621760}]
I20260928 21:52:13.387583 18964 mvcc.cc:205] Tried to move back new op lower bound from 7334430037541621760 to 7334430037501775872. Current Snapshot: MvccSnapshot[applied={T|T < 7334430037541621760}]
I20260928 21:52:13.387396 18965 mvcc.cc:205] Tried to move back new op lower bound from 7334430037541621760 to 7334430037501775872. Current Snapshot: MvccSnapshot[applied={T|T < 7334430037541621760}]
I20260928 21:52:13.389144 18477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:52:13.389219 18477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:52:13.389847 18477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Sending DeleteTablet for 3 replicas of tablet 532611d7ad144308ab745f2bc2c521bb
I20260928 21:52:13.390148 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 18539
I20260928 21:52:13.390221 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 532611d7ad144308ab745f2bc2c521bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:52:13 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:58166
I20260928 21:52:13.390213 18870 tablet_service.cc:1514] Processing DeleteTablet for tablet 532611d7ad144308ab745f2bc2c521bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:52:13 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:47326
I20260928 21:52:13.390599 18736 tablet_service.cc:1514] Processing DeleteTablet for tablet 532611d7ad144308ab745f2bc2c521bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:52:13 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:55266
I20260928 21:52:13.392580 18973 tablet_replica.cc:330] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: stopping tablet replica
I20260928 21:52:13.392709 18974 tablet_replica.cc:330] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: stopping tablet replica
I20260928 21:52:13.392804 18974 raft_consensus.cc:2229] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:52:13.392896 18974 raft_consensus.cc:2258] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:52:13.393045 18973 raft_consensus.cc:2229] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:52:13.393188 18973 raft_consensus.cc:2258] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:52:13.393345 18974 ts_tablet_manager.cc:1905] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:52:13.393675 18973 ts_tablet_manager.cc:1905] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:52:13.395030 18974 ts_tablet_manager.cc:1918] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:52:13.395156 18974 log.cc:1196] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/532611d7ad144308ab745f2bc2c521bb
I20260928 21:52:13.395294 18973 ts_tablet_manager.cc:1918] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:52:13.395440 18973 log.cc:1196] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/532611d7ad144308ab745f2bc2c521bb
I20260928 21:52:13.395465 18974 ts_tablet_manager.cc:1939] T 532611d7ad144308ab745f2bc2c521bb P ee397f845f8047979795c00290b51aee: Deleting consensus metadata
W20260928 21:52:13.395671 18463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:35837 (error 108)
I20260928 21:52:13.395816 18973 ts_tablet_manager.cc:1939] T 532611d7ad144308ab745f2bc2c521bb P 3c8f1afe27ff436e99f9cda9b87d2e40: Deleting consensus metadata
I20260928 21:52:13.395925 18461 catalog_manager.cc:4841] TS ee397f845f8047979795c00290b51aee (127.3.0.194:37769): tablet 532611d7ad144308ab745f2bc2c521bb (table test-table [id=f9fb54df387f471297a0d37e9f72fc08]) successfully deleted
I20260928 21:52:13.396186 18461 catalog_manager.cc:4841] TS 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381): tablet 532611d7ad144308ab745f2bc2c521bb (table test-table [id=f9fb54df387f471297a0d37e9f72fc08]) successfully deleted
I20260928 21:52:13.396283 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 18673
I20260928 21:52:13.397137 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:13.396994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:35837 (local address 127.0.0.1:58168)
0928 21:52:13.397065 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:13.397110 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 21:52:13.397265 18463 catalog_manager.cc:4584] TS 0b755691f36c4e689d712cb193612fcf (127.3.0.193:35837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 532611d7ad144308ab745f2bc2c521bb: Network error: Client connection negotiation failed: client connection to 127.3.0.193:35837: connect: Connection refused (error 111)
I20260928 21:52:13.401468 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 18807
I20260928 21:52:13.405426 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 18445
I20260928 21:52:33.409909 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38227
--webserver_interface=127.3.0.254
--webserver_port=33691
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:52:33.487398 18976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:33.487593 18976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:33.487613 18976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:33.488797 18976 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:52:33.488835 18976 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:52:33.488847 18976 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:52:33.488858 18976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:52:33.488883 18976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:33.488905 18976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:33.488925 18976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:52:33.488946 18976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:52:33.490432 18976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:38227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:33.490655 18976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:33.490861 18976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:52:33.498869 18983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:33.498951 18982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:33.498981 18976 server_base.cc:1034] running on GCE node
W20260928 21:52:33.498869 18985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:33.499300 18976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:33.499607 18976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:33.500743 18976 hybrid_clock.cc:648] HybridClock initialized: now 1790632353500728 us; error 29 us; skew 500 ppm
Sep 28 21:52:33 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:33.503127 18976 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:52:33.503556 18976 webserver.cc:460] Webserver started at http://127.3.0.254:33691/ using document root <none> and password file <none>
I20260928 21:52:33.503749 18976 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:33.503801 18976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:33.505087 18976 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:33.505648 18992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:33.505812 18976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:52:33.505893 18976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "a347157b6f7b4abe8f144601793d40df"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "520d331bb7184312f3186b5cd45f35f0"
server_key_iv: "3a2a503955b3b5e7cfe8d13347ae3753"
server_key_version: "encryptionkey@0"
I20260928 21:52:33.506259 18976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:33.516981 18976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:33.518703 18976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:33.518810 18976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:33.522482 18976 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:38227
I20260928 21:52:33.522521 19044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:38227 every 8 connection(s)
I20260928 21:52:33.522848 18976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:52:33.524252 19045 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:52:33.524276 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 18976
I20260928 21:52:33.524567 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:35837
--local_ip_for_outbound_sockets=127.3.0.193
--tserver_master_addrs=127.3.0.254:38227
--webserver_port=35741
--webserver_interface=127.3.0.193
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:52:33.525009 19045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Bootstrap starting.
I20260928 21:52:33.532958 19045 log.cc:824] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Log is configured to *not* fsync() on all Append() calls
I20260928 21:52:33.534688 19045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:52:33.534924 19045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Bootstrap complete.
I20260928 21:52:33.537011 19045 raft_consensus.cc:348] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:33.537230 19045 raft_consensus.cc:729] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a347157b6f7b4abe8f144601793d40df, State: Initialized, Role: FOLLOWER
I20260928 21:52:33.537372 19045 consensus_queue.cc:260] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:33.537442 19045 raft_consensus.cc:388] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:52:33.537493 19045 raft_consensus.cc:482] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:52:33.537533 19045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:52:33.538475 19045 raft_consensus.cc:504] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:33.538568 19045 leader_election.cc:302] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a347157b6f7b4abe8f144601793d40df; no voters:
I20260928 21:52:33.538713 19045 leader_election.cc:288] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:52:33.538760 19049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:52:33.538916 19049 raft_consensus.cc:686] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 2 LEADER]: Becoming Leader. State: Replica: a347157b6f7b4abe8f144601793d40df, State: Running, Role: LEADER
I20260928 21:52:33.538980 19045 sys_catalog.cc:564] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:52:33.539196 19049 consensus_queue.cc:237] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:33.539953 19050 sys_catalog.cc:455] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: SysCatalogTable state changed. Reason: New leader a347157b6f7b4abe8f144601793d40df. Latest consensus state: current_term: 2 leader_uuid: "a347157b6f7b4abe8f144601793d40df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } } }
I20260928 21:52:33.540030 19050 sys_catalog.cc:458] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: This master's current role is: LEADER
I20260928 21:52:33.540294 19062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:52:33.540557 19051 sys_catalog.cc:455] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a347157b6f7b4abe8f144601793d40df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } } }
I20260928 21:52:33.540637 19051 sys_catalog.cc:458] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: This master's current role is: LEADER
I20260928 21:52:33.541183 19062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:52:33.541283 19062 catalog_manager.cc:1259] Loaded cluster ID: de27b1c2e9b24b6894d387e460a99b39
I20260928 21:52:33.541316 19062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:52:33.542099 19062 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:52:33.542266 19062 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Loaded TSK: 0
I20260928 21:52:33.551688 19062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Generated new TSK 1
I20260928 21:52:33.551774 19062 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:52:33.609498 19047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:33.609716 19047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:33.609745 19047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:33.610846 19047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:33.610884 19047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:33.610918 19047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:52:33.612298 19047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:35837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=35741
--tserver_master_addrs=127.3.0.254:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:33.612521 19047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:33.612712 19047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:52:33.617137 19075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:33.617151 19073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:33.617342 19047 server_base.cc:1034] running on GCE node
W20260928 21:52:33.617169 19072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:33.617600 19047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:33.617811 19047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:33.618940 19047 hybrid_clock.cc:648] HybridClock initialized: now 1790632353618928 us; error 31 us; skew 500 ppm
Sep 28 21:52:33 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:33.621336 19047 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:52:33.621747 19047 webserver.cc:460] Webserver started at http://127.3.0.193:35741/ using document root <none> and password file <none>
I20260928 21:52:33.621938 19047 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:33.621995 19047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:33.623209 19047 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:52:33.623786 19082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:33.623939 19047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:52:33.624015 19047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "0b755691f36c4e689d712cb193612fcf"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "0c8e166ff84aef515d1899a6b606ac05"
server_key_iv: "759efaaffb76fb7aba219f3a69f18bbf"
server_key_version: "encryptionkey@0"
I20260928 21:52:33.624323 19047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:33.639576 19047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:33.641708 19047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:33.641815 19047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:33.642033 19047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:52:33.642462 19089 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:52:33.643774 19047 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:52:33.643821 19047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260928 21:52:33.643842 19047 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:52:33.644428 19047 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:52:33.644503 19047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:33.644563 19089 tablet_bootstrap.cc:492] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Bootstrap starting.
I20260928 21:52:33.651487 19047 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:35837
I20260928 21:52:33.651561 19196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:35837 every 8 connection(s)
I20260928 21:52:33.651942 19047 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:52:33.655242 19089 log.cc:824] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Log is configured to *not* fsync() on all Append() calls
I20260928 21:52:33.657382 19089 tablet_bootstrap.cc:492] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:52:33.657670 19089 tablet_bootstrap.cc:492] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Bootstrap complete.
I20260928 21:52:33.658710 19089 ts_tablet_manager.cc:1397] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260928 21:52:33.659085 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 19047
I20260928 21:52:33.659348 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:37769
--local_ip_for_outbound_sockets=127.3.0.194
--tserver_master_addrs=127.3.0.254:38227
--webserver_port=39567
--webserver_interface=127.3.0.194
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:52:33.660032 19089 raft_consensus.cc:348] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:33.660307 19089 raft_consensus.cc:729] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b755691f36c4e689d712cb193612fcf, State: Initialized, Role: FOLLOWER
I20260928 21:52:33.660470 19089 consensus_queue.cc:260] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } }
I20260928 21:52:33.660739 19089 ts_tablet_manager.cc:1428] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 28 21:52:33 dist-test-slave-jrck krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:52:33.667466 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.653074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:56891 (local address 127.3.0.254:38227)
0928 21:52:33.656588 (+ 3514us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:33.656598 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:33.656643 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:52:33.656784 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:33.656791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:33.656845 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:33.656978 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:33.657796 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.659133 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.660574 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.661156 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.661260 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.663790 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:33.663795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:33.663795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:33.663807 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:33.665689 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:33.666139 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:33.666141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:33.666142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:33.666238 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:33.666494 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:33.666494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:33.666494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:33.666756 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:33.666832 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:33.667147 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 21:52:33.667274 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3457,"thread_start_us":114,"threads_started":1}
I20260928 21:52:33.667470 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.652695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:56891)
0928 21:52:33.653055 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:33.653100 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:33.653669 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:33.657036 (+ 3367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:33.657041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:33.657384 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:52:33.657659 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.657666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.659260 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.659261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.660444 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.660446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.661229 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.661230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.661365 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.661941 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:52:33.661952 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:52:33.663690 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:52:33.665768 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:33.665776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:33.665778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:33.666065 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:33.666299 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:33.666299 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:33.666300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:33.666433 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:33.666820 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:52:33.666826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:52:33.666983 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 21:52:33.667097 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:52:33.667272 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":88,"threads_started":1}
I20260928 21:52:33.668546 19197 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38227
I20260928 21:52:33.668644 19197 heartbeater.cc:464] Registering TS with master...
I20260928 21:52:33.668823 19197 heartbeater.cc:511] Master 127.3.0.254:38227 requested a full tablet report, sending...
I20260928 21:52:33.669623 19009 ts_manager.cc:194] Registered new tserver with Master: 0b755691f36c4e689d712cb193612fcf (127.3.0.193:35837)
I20260928 21:52:33.670675 19009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:56891
I20260928 21:52:33.672029 19197 heartbeater.cc:503] Master 127.3.0.254:38227 was elected leader, sending a full tablet report...
Sep 28 21:52:33 dist-test-slave-jrck krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.193@KRBTEST.COM
I20260928 21:52:33.678515 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.670433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.3.0.193:35837)
0928 21:52:33.670569 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:33.670574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:33.671131 (+ 557us) server_negotiation.cc:407] Connection header received
0928 21:52:33.671165 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:33.671167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:33.671180 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:33.671229 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:33.671936 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.673071 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.673844 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.674138 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.674192 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.676148 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:33.676155 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:33.676155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:33.676161 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:33.677504 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:33.677884 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:33.677885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:33.677886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:33.677933 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:33.678087 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:33.678088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:33.678088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:33.678190 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:33.678246 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:33.678396 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:52:33.678454 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260928 21:52:33.678572 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.670216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:35837 (local address 127.0.0.1:42504)
0928 21:52:33.670971 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:33.671000 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:33.671063 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:33.671253 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:33.671258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:33.671537 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:52:33.671591 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.671593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.673164 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.673165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.673770 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.673772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.674213 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.674214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.674281 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.674678 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:52:33.674682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:52:33.676077 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:52:33.677577 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:33.677583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:33.677585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:33.677820 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:33.677982 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:33.677982 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:33.677983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:33.678041 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:33.678241 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:52:33.678246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:52:33.678293 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:52:33.678370 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:52:33.678476 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":75,"threads_started":1}
I20260928 21:52:33.678925 19113 tablet_service.cc:1514] Processing DeleteTablet for tablet 532611d7ad144308ab745f2bc2c521bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:52:13 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:42504
I20260928 21:52:33.679478 19206 tablet_replica.cc:330] stopping tablet replica
I20260928 21:52:33.679572 19206 raft_consensus.cc:2229] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:52:33.679636 19206 raft_consensus.cc:2258] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:52:33.679908 19206 ts_tablet_manager.cc:1905] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:52:33.681958 19206 ts_tablet_manager.cc:1918] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:52:33.682020 19206 log.cc:1196] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal/wals/532611d7ad144308ab745f2bc2c521bb
I20260928 21:52:33.682224 19206 ts_tablet_manager.cc:1939] T 532611d7ad144308ab745f2bc2c521bb P 0b755691f36c4e689d712cb193612fcf: Deleting consensus metadata
I20260928 21:52:33.682654 18995 catalog_manager.cc:4841] TS 0b755691f36c4e689d712cb193612fcf (127.3.0.193:35837): tablet 532611d7ad144308ab745f2bc2c521bb (table test-table [id=f9fb54df387f471297a0d37e9f72fc08]) successfully deleted
W20260928 21:52:33.742810 19203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:33.743014 19203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:33.743036 19203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:33.744158 19203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:33.744196 19203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:33.744221 19203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:52:33.745631 19203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:37769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=39567
--tserver_master_addrs=127.3.0.254:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:33.745828 19203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:33.746001 19203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:52:33.750568 19214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:33.750710 19203 server_base.cc:1034] running on GCE node
W20260928 21:52:33.750557 19212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:33.750557 19211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:33.751008 19203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:33.751209 19203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:33.752334 19203 hybrid_clock.cc:648] HybridClock initialized: now 1790632353752327 us; error 34 us; skew 500 ppm
Sep 28 21:52:33 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:33.754683 19203 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:52:33.755086 19203 webserver.cc:460] Webserver started at http://127.3.0.194:39567/ using document root <none> and password file <none>
I20260928 21:52:33.755314 19203 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:33.755373 19203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:33.756621 19203 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:33.757125 19221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:33.757325 19203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:33.757408 19203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "ee397f845f8047979795c00290b51aee"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "3dc548adc42e580fbcc139c2044c0a65"
server_key_iv: "2a5903cccb4f36af0b329930cae36f24"
server_key_version: "encryptionkey@0"
I20260928 21:52:33.757757 19203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:33.774534 19203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:33.776261 19203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:33.776381 19203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:33.776598 19203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:52:33.776878 19203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:52:33.776909 19203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:33.776930 19203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:52:33.776954 19203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:33.782459 19203 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:37769
I20260928 21:52:33.782531 19334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:37769 every 8 connection(s)
I20260928 21:52:33.782826 19203 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:52:33.784696 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 19203
I20260928 21:52:33.784972 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:37381
--local_ip_for_outbound_sockets=127.3.0.195
--tserver_master_addrs=127.3.0.254:38227
--webserver_port=40587
--webserver_interface=127.3.0.195
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:52:33 dist-test-slave-jrck krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:52:33.792165 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.783456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:59637 (local address 127.3.0.254:38227)
0928 21:52:33.783499 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:33.783501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:33.783715 (+ 214us) server_negotiation.cc:407] Connection header received
0928 21:52:33.784174 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:33.784175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:33.784189 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:33.784242 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:33.785391 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.785909 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.786955 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.787248 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.787318 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.789543 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:33.789545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:33.789545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:33.789551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:33.790809 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:33.791252 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:33.791252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:33.791253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:33.791298 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:33.791555 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:33.791555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:33.791555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:33.791650 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:33.791694 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:33.792012 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 21:52:33.792077 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:52:33.792412 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.783437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:59637)
0928 21:52:33.783590 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:33.783646 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:33.784099 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:33.784305 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:33.784308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:33.785028 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:52:33.785302 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.785310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.785999 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.786000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.786870 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.786872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.787322 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.787322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.787420 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.787871 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:52:33.787878 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:52:33.789455 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:52:33.790883 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:33.790891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:33.790893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:33.791186 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:33.791354 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:33.791354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:33.791355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:33.791497 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:33.791710 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:52:33.791716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:52:33.791885 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 21:52:33.792005 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 21:52:33.792171 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260928 21:52:33.792943 19335 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38227
I20260928 21:52:33.793028 19335 heartbeater.cc:464] Registering TS with master...
I20260928 21:52:33.793198 19335 heartbeater.cc:511] Master 127.3.0.254:38227 requested a full tablet report, sending...
I20260928 21:52:33.793524 19009 ts_manager.cc:194] Registered new tserver with Master: ee397f845f8047979795c00290b51aee (127.3.0.194:37769)
I20260928 21:52:33.793867 19009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:59637
W20260928 21:52:33.864404 19338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:33.864631 19338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:33.864661 19338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:33.865790 19338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:33.865831 19338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:33.865864 19338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:52:33.867276 19338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:37381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=40587
--tserver_master_addrs=127.3.0.254:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:33.867496 19338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:33.867690 19338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:52:33.872224 19343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:33.872219 19344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:33.872243 19338 server_base.cc:1034] running on GCE node
W20260928 21:52:33.872222 19346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:33.872618 19338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:33.872851 19338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:33.873986 19338 hybrid_clock.cc:648] HybridClock initialized: now 1790632353873974 us; error 27 us; skew 500 ppm
Sep 28 21:52:33 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:33.876323 19338 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:52:33.876735 19338 webserver.cc:460] Webserver started at http://127.3.0.195:40587/ using document root <none> and password file <none>
I20260928 21:52:33.876940 19338 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:33.876993 19338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:33.878206 19338 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:33.878724 19353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:33.878844 19338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:33.878907 19338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "3c8f1afe27ff436e99f9cda9b87d2e40"
format_stamp: "Formatted at 2026-09-28 21:52:13 on dist-test-slave-jrck"
server_key: "7c54591c844e036061c2fc97476ac422"
server_key_iv: "1eb152d7c4ff077ba5807c5a6ce91919"
server_key_version: "encryptionkey@0"
I20260928 21:52:33.879253 19338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:33.887552 19338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:33.889556 19338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:33.889675 19338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:33.889861 19338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:52:33.890121 19338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:52:33.890151 19338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:33.890170 19338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:52:33.890182 19338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:33.895470 19338 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:37381
I20260928 21:52:33.895531 19466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:37381 every 8 connection(s)
I20260928 21:52:33.895825 19338 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:52:33.899466 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 19338
Sep 28 21:52:33 dist-test-slave-jrck krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:52:33.905189 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.896513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:49501 (local address 127.3.0.254:38227)
0928 21:52:33.896559 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:33.896561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:33.896748 (+ 187us) server_negotiation.cc:407] Connection header received
0928 21:52:33.897264 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:33.897266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:33.897279 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:33.897330 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:33.898230 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.898722 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.899738 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.900008 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.900073 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.902500 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:33.902501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:33.902501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:33.902507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:33.903811 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:33.904252 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:33.904253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:33.904253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:33.904304 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:33.904556 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:33.904557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:33.904558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:33.904683 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:33.904736 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:33.905037 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 21:52:33.905126 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:52:33.905400 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.896496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:49501)
0928 21:52:33.896633 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:33.896667 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:33.897152 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:33.897399 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:33.897403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:33.897949 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:52:33.898134 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.898139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.898804 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.898805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.899660 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.899663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.900092 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.900093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.900262 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.900779 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:52:33.900789 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:52:33.902417 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:52:33.903879 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:33.903886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:33.903889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:33.904180 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:33.904362 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:33.904363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:33.904363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:33.904509 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:33.904740 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:52:33.904746 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:52:33.904908 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 21:52:33.905015 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 21:52:33.905175 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260928 21:52:33.905886 19467 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38227
I20260928 21:52:33.905989 19467 heartbeater.cc:464] Registering TS with master...
I20260928 21:52:33.906226 19467 heartbeater.cc:511] Master 127.3.0.254:38227 requested a full tablet report, sending...
I20260928 21:52:33.906324 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.899833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.3.0.254:38227)
0928 21:52:33.899944 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:33.899946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:33.900020 (+ 74us) server_negotiation.cc:407] Connection header received
0928 21:52:33.900082 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:33.900082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:33.900094 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:33.900162 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:33.900804 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.901497 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.902450 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.902653 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.902709 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.903743 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:33.903744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:33.903744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:33.903751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:33.905244 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:33.905604 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:33.905605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:33.905606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:33.905661 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:33.905827 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:33.905828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:33.905828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:33.905957 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:33.906122 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:33.906160 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:52:33.906242 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260928 21:52:33.906641 19009 ts_manager.cc:194] Registered new tserver with Master: 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381)
I20260928 21:52:33.907008 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:52:33.907068 19009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:49501
I20260928 21:52:33.909526 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.907301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.3.0.254:38227)
0928 21:52:33.907331 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:33.907332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:33.907470 (+ 138us) server_negotiation.cc:407] Connection header received
0928 21:52:33.907530 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:33.907530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:33.907557 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:33.907610 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:52:33.907718 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.908179 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.908953 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.909127 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.909170 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.909286 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:52:33.909384 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:52:33.909432 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:33.909464 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 21:52:33.909480 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260928 21:52:33.909744 19009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:52:33.911245 19009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:52:33.913108 19113 tablet_service.cc:1467] Processing CreateTablet for tablet f1db8ad904b444eb8832e20f7a8fb7da (DEFAULT_TABLE table=test-table [id=b7cd4fa7f0524e55b7d6510ec3a8be89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:52:33.913300 19113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1db8ad904b444eb8832e20f7a8fb7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:52:33.914572 19089 tablet_bootstrap.cc:492] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Bootstrap starting.
I20260928 21:52:33.914896 19089 tablet_bootstrap.cc:654] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Neither blocks nor log segments found. Creating new log.
I20260928 21:52:33.915431 19089 tablet_bootstrap.cc:492] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: No bootstrap required, opened a new log
I20260928 21:52:33.915477 19089 ts_tablet_manager.cc:1397] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:33.915793 19089 raft_consensus.cc:348] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:33.915874 19089 raft_consensus.cc:374] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:52:33.915908 19089 raft_consensus.cc:729] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b755691f36c4e689d712cb193612fcf, State: Initialized, Role: FOLLOWER
I20260928 21:52:33.915899 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.912819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:37381 (local address 127.0.0.1:33426)
0928 21:52:33.912859 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:33.912869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:33.912903 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:33.913187 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:33.913187 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:33.913189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:52:33.913224 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.913225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.914046 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.914047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.914906 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.914908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.915649 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.915650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.915723 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.915741 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:52:33.915790 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:52:33.915835 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 21:52:33.915913 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.912877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.3.0.195:37381)
0928 21:52:33.913036 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:33.913040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:33.913050 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:52:33.913081 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:33.913083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:33.913097 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:33.913179 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:52:33.913291 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.913988 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.914992 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.915574 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.915636 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.915692 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:33.915810 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 21:52:33.915846 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260928 21:52:33.915947 19089 consensus_queue.cc:260] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:33.916031 19089 ts_tablet_manager.cc:1428] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:52:33.916765 19401 tablet_service.cc:1467] Processing CreateTablet for tablet f1db8ad904b444eb8832e20f7a8fb7da (DEFAULT_TABLE table=test-table [id=b7cd4fa7f0524e55b7d6510ec3a8be89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:52:33.917069 19401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1db8ad904b444eb8832e20f7a8fb7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:52:33.918110 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.912903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:37769 (local address 127.0.0.1:58578)
0928 21:52:33.914617 (+ 1714us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:33.914628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:33.914682 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:33.914898 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:33.914899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:33.914900 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:52:33.914932 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.914933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.916074 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.916075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.917135 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:33.917137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.917961 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.917962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:33.918012 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.918014 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:52:33.918038 (+ 24us) client_negotiation.cc:240] Negotiation successful
0928 21:52:33.918064 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1696,"thread_start_us":90,"threads_started":1}
I20260928 21:52:33.918273 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:33.913135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.3.0.194:37769)
0928 21:52:33.913411 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:33.913416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:33.914752 (+ 1336us) server_negotiation.cc:407] Connection header received
0928 21:52:33.914800 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:33.914803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:33.914817 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:33.914893 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:52:33.914986 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.915964 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.917205 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:33.917889 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:33.917966 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:33.918025 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:33.918060 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 21:52:33.918099 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":92,"threads_started":1}
I20260928 21:52:33.918916 19269 tablet_service.cc:1467] Processing CreateTablet for tablet f1db8ad904b444eb8832e20f7a8fb7da (DEFAULT_TABLE table=test-table [id=b7cd4fa7f0524e55b7d6510ec3a8be89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:52:33.919212 19269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1db8ad904b444eb8832e20f7a8fb7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:52:33.920332 19477 tablet_bootstrap.cc:492] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: Bootstrap starting.
I20260928 21:52:33.920898 19477 tablet_bootstrap.cc:654] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: Neither blocks nor log segments found. Creating new log.
I20260928 21:52:33.921341 19477 log.cc:824] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: Log is configured to *not* fsync() on all Append() calls
I20260928 21:52:33.922071 19477 tablet_bootstrap.cc:492] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: No bootstrap required, opened a new log
I20260928 21:52:33.922149 19479 tablet_bootstrap.cc:492] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: Bootstrap starting.
I20260928 21:52:33.922130 19477 ts_tablet_manager.cc:1397] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:52:33.922662 19479 tablet_bootstrap.cc:654] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: Neither blocks nor log segments found. Creating new log.
I20260928 21:52:33.923105 19479 log.cc:824] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: Log is configured to *not* fsync() on all Append() calls
I20260928 21:52:33.923672 19479 tablet_bootstrap.cc:492] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: No bootstrap required, opened a new log
I20260928 21:52:33.923597 19477 raft_consensus.cc:348] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:33.923722 19479 ts_tablet_manager.cc:1397] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:52:33.923722 19477 raft_consensus.cc:374] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:52:33.923754 19477 raft_consensus.cc:729] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8f1afe27ff436e99f9cda9b87d2e40, State: Initialized, Role: FOLLOWER
I20260928 21:52:33.923862 19477 consensus_queue.cc:260] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:33.924093 19477 ts_tablet_manager.cc:1428] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:52:33.924125 19467 heartbeater.cc:503] Master 127.3.0.254:38227 was elected leader, sending a full tablet report...
I20260928 21:52:33.925457 19479 raft_consensus.cc:348] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:33.925582 19479 raft_consensus.cc:374] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:52:33.925602 19479 raft_consensus.cc:729] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee397f845f8047979795c00290b51aee, State: Initialized, Role: FOLLOWER
I20260928 21:52:33.925684 19479 consensus_queue.cc:260] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:33.925902 19479 ts_tablet_manager.cc:1428] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:52:33.925961 19335 heartbeater.cc:503] Master 127.3.0.254:38227 was elected leader, sending a full tablet report...
W20260928 21:52:34.033407 19336 tablet.cc:2369] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:52:34.146368 19468 tablet.cc:2369] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:52:34.153113 19198 tablet.cc:2369] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:52:34.178958 19202 raft_consensus.cc:482] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:52:34.179085 19202 raft_consensus.cc:504] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:34.179369 19202 leader_election.cc:288] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381), ee397f845f8047979795c00290b51aee (127.3.0.194:37769)
I20260928 21:52:34.182391 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:34.179818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:56793 (local address 127.3.0.194:37769)
0928 21:52:34.179871 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:34.179873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:34.179884 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:52:34.179966 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:34.179967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:34.179979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:34.180030 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:52:34.180149 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.180624 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.181644 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.182074 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.182135 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:34.182147 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:34.182289 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 21:52:34.182326 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:52:34.182430 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:34.179627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:55559 (local address 127.3.0.195:37381)
0928 21:52:34.179678 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:34.179680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:34.179751 (+ 71us) server_negotiation.cc:407] Connection header received
0928 21:52:34.179899 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:34.179900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:34.179914 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:34.179966 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:52:34.180093 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.180629 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.181640 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.182124 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.182186 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:34.182196 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:34.182325 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 21:52:34.182362 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:52:34.182441 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:34.179744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:37769 (local address 127.3.0.193:56793)
0928 21:52:34.179824 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:34.179834 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:34.179898 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:34.180040 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:34.180041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:34.180043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:52:34.180089 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:34.180090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.180695 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.180696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:34.181572 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:34.181575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.182144 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.182145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:34.182208 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:34.182231 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:52:34.182277 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:52:34.182320 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":33,"thread_start_us":37,"threads_started":1}
I20260928 21:52:34.182451 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:34.179564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:37381 (local address 127.3.0.193:55559)
0928 21:52:34.179680 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:34.179692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:34.179775 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:34.179971 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:34.179972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:34.179974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:52:34.180034 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:34.180036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.180714 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.180715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:34.181552 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:34.181554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.182196 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.182197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:34.182267 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:34.182271 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:52:34.182315 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:52:34.182354 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":36,"thread_start_us":57,"threads_started":1}
I20260928 21:52:34.182817 19421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1db8ad904b444eb8832e20f7a8fb7da" candidate_uuid: "0b755691f36c4e689d712cb193612fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" is_pre_election: true
I20260928 21:52:34.182890 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1db8ad904b444eb8832e20f7a8fb7da" candidate_uuid: "0b755691f36c4e689d712cb193612fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee397f845f8047979795c00290b51aee" is_pre_election: true
I20260928 21:52:34.182998 19421 raft_consensus.cc:2454] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b755691f36c4e689d712cb193612fcf in term 0.
I20260928 21:52:34.183001 19289 raft_consensus.cc:2454] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b755691f36c4e689d712cb193612fcf in term 0.
I20260928 21:52:34.183187 19083 leader_election.cc:302] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b755691f36c4e689d712cb193612fcf, 3c8f1afe27ff436e99f9cda9b87d2e40; no voters:
I20260928 21:52:34.183290 19202 raft_consensus.cc:2781] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:52:34.183358 19202 raft_consensus.cc:482] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:52:34.183374 19202 raft_consensus.cc:3037] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:52:34.184026 19202 raft_consensus.cc:504] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:34.184139 19202 leader_election.cc:288] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 1 election: Requested vote from peers 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381), ee397f845f8047979795c00290b51aee (127.3.0.194:37769)
I20260928 21:52:34.184314 19289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1db8ad904b444eb8832e20f7a8fb7da" candidate_uuid: "0b755691f36c4e689d712cb193612fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee397f845f8047979795c00290b51aee"
I20260928 21:52:34.184338 19421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1db8ad904b444eb8832e20f7a8fb7da" candidate_uuid: "0b755691f36c4e689d712cb193612fcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40"
I20260928 21:52:34.184397 19421 raft_consensus.cc:3037] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:52:34.184391 19289 raft_consensus.cc:3037] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:52:34.185072 19421 raft_consensus.cc:2454] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b755691f36c4e689d712cb193612fcf in term 1.
I20260928 21:52:34.185123 19289 raft_consensus.cc:2454] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b755691f36c4e689d712cb193612fcf in term 1.
I20260928 21:52:34.185256 19083 leader_election.cc:302] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b755691f36c4e689d712cb193612fcf, ee397f845f8047979795c00290b51aee; no voters:
I20260928 21:52:34.185346 19202 raft_consensus.cc:2781] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:52:34.185460 19202 raft_consensus.cc:686] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 LEADER]: Becoming Leader. State: Replica: 0b755691f36c4e689d712cb193612fcf, State: Running, Role: LEADER
I20260928 21:52:34.185527 19202 consensus_queue.cc:237] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:34.186259 19009 catalog_manager.cc:5495] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b755691f36c4e689d712cb193612fcf (127.3.0.193). New cstate: current_term: 1 leader_uuid: "0b755691f36c4e689d712cb193612fcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } health_report { overall_health: HEALTHY } } }
I20260928 21:52:34.201478 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:34.199375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.3.0.193:35837)
0928 21:52:34.199498 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:34.199500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:34.199507 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:52:34.199540 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:34.199540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:34.199555 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:34.199621 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:52:34.199719 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.200207 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.200948 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.201130 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.201177 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:34.201278 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:52:34.201324 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 21:52:34.201420 (+ 96us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42512: token expired
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
W20260928 21:52:34.201542 19488 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42512: token expired
I20260928 21:52:34.206594 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:34.201844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.3.0.254:38227)
0928 21:52:34.201885 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:34.201887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:34.201894 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:52:34.201912 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:34.201912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:34.201920 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:34.201958 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:34.202373 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.202817 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.203406 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.203569 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.203607 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:34.204695 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:34.204696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:34.204696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:34.204700 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:34.205826 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:34.206093 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:34.206094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:34.206094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:34.206135 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:34.206260 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:34.206260 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:34.206260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:34.206344 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:34.206408 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:34.206474 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 21:52:34.206532 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:52:34.209398 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:34.207321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.3.0.193:35837)
0928 21:52:34.207355 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:34.207357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:34.207365 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:52:34.207409 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:34.207410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:34.207418 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:34.207460 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:52:34.207534 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.207989 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.208755 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:34.208940 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:34.208990 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:34.209101 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:52:34.209194 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:52:34.209253 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:34.209310 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:52:34.209329 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:52:34.210873 19289 raft_consensus.cc:1261] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee [term 1 FOLLOWER]: Refusing update from remote peer 0b755691f36c4e689d712cb193612fcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:52:34.210872 19421 raft_consensus.cc:1261] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 1 FOLLOWER]: Refusing update from remote peer 0b755691f36c4e689d712cb193612fcf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:52:34.211191 19202 consensus_queue.cc:1035] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:52:34.211338 19485 consensus_queue.cc:1035] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:52:34.213496 19489 mvcc.cc:205] Tried to move back new op lower bound from 7334430122845315072 to 7334430122744623104. Current Snapshot: MvccSnapshot[applied={T|T < 7334430122845315072}]
I20260928 21:52:34.213672 19491 mvcc.cc:205] Tried to move back new op lower bound from 7334430122845315072 to 7334430122744623104. Current Snapshot: MvccSnapshot[applied={T|T < 7334430122845315072}]
I20260928 21:52:34.215956 19008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:52:34.216044 19008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:52:34.216715 19008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Sending DeleteTablet for 3 replicas of tablet f1db8ad904b444eb8832e20f7a8fb7da
I20260928 21:52:34.216959 19113 tablet_service.cc:1514] Processing DeleteTablet for tablet f1db8ad904b444eb8832e20f7a8fb7da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:52:34 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:42504
I20260928 21:52:34.216996 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 19047
I20260928 21:52:34.217046 19401 tablet_service.cc:1514] Processing DeleteTablet for tablet f1db8ad904b444eb8832e20f7a8fb7da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:52:34 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:33426
I20260928 21:52:34.217100 19269 tablet_service.cc:1514] Processing DeleteTablet for tablet f1db8ad904b444eb8832e20f7a8fb7da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:52:34 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:58578
I20260928 21:52:34.217326 19500 tablet_replica.cc:330] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: stopping tablet replica
I20260928 21:52:34.217478 19500 raft_consensus.cc:2229] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:52:34.217555 19499 tablet_replica.cc:330] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: stopping tablet replica
I20260928 21:52:34.217593 19500 raft_consensus.cc:2258] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:52:34.217689 19499 raft_consensus.cc:2229] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:52:34.217821 19499 raft_consensus.cc:2258] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:52:34.217993 19500 ts_tablet_manager.cc:1905] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:52:34.218258 19499 ts_tablet_manager.cc:1905] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:52:34.219547 19500 ts_tablet_manager.cc:1918] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:52:34.219609 19500 log.cc:1196] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/f1db8ad904b444eb8832e20f7a8fb7da
I20260928 21:52:34.219726 19499 ts_tablet_manager.cc:1918] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:52:34.219795 19499 log.cc:1196] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/f1db8ad904b444eb8832e20f7a8fb7da
I20260928 21:52:34.219921 19500 ts_tablet_manager.cc:1939] T f1db8ad904b444eb8832e20f7a8fb7da P ee397f845f8047979795c00290b51aee: Deleting consensus metadata
I20260928 21:52:34.220083 19499 ts_tablet_manager.cc:1939] T f1db8ad904b444eb8832e20f7a8fb7da P 3c8f1afe27ff436e99f9cda9b87d2e40: Deleting consensus metadata
I20260928 21:52:34.220299 18993 catalog_manager.cc:4841] TS ee397f845f8047979795c00290b51aee (127.3.0.194:37769): tablet f1db8ad904b444eb8832e20f7a8fb7da (table test-table [id=b7cd4fa7f0524e55b7d6510ec3a8be89]) successfully deleted
I20260928 21:52:34.220404 18993 catalog_manager.cc:4841] TS 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381): tablet f1db8ad904b444eb8832e20f7a8fb7da (table test-table [id=b7cd4fa7f0524e55b7d6510ec3a8be89]) successfully deleted
W20260928 21:52:34.222836 18995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:35837 (error 108)
I20260928 21:52:34.223250 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:34.223130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:35837 (local address 127.0.0.1:42534)
0928 21:52:34.223179 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:34.223213 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:35837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 21:52:34.223357 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 19203
W20260928 21:52:34.223376 18995 catalog_manager.cc:4584] TS 0b755691f36c4e689d712cb193612fcf (127.3.0.193:35837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1db8ad904b444eb8832e20f7a8fb7da: Network error: Client connection negotiation failed: client connection to 127.3.0.193:35837: connect: Connection refused (error 111)
I20260928 21:52:34.227504 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 19338
I20260928 21:52:34.231523 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 18976
I20260928 21:52:54.236161 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38227
--webserver_interface=127.3.0.254
--webserver_port=33691
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:38227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:52:54.315830 19502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:54.316058 19502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:54.316083 19502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:54.317355 19502 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:52:54.317397 19502 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:52:54.317408 19502 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:52:54.317425 19502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:52:54.317451 19502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:54.317476 19502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:54.317499 19502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:52:54.317521 19502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:52:54.319048 19502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:38227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:54.319294 19502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:54.319521 19502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:52:54.324435 19507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:54.324477 19508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:54.324517 19502 server_base.cc:1034] running on GCE node
W20260928 21:52:54.324448 19510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:54.324887 19502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:54.325206 19502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:54.326382 19502 hybrid_clock.cc:648] HybridClock initialized: now 1790632374326360 us; error 70 us; skew 500 ppm
Sep 28 21:52:54 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632374, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:54.329149 19502 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:52:54.329666 19502 webserver.cc:460] Webserver started at http://127.3.0.254:33691/ using document root <none> and password file <none>
I20260928 21:52:54.329901 19502 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:54.329954 19502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:54.331499 19502 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 21:52:54.332348 19517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:54.332556 19502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:54.332643 19502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "a347157b6f7b4abe8f144601793d40df"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "520d331bb7184312f3186b5cd45f35f0"
server_key_iv: "3a2a503955b3b5e7cfe8d13347ae3753"
server_key_version: "encryptionkey@0"
I20260928 21:52:54.333043 19502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:54.357401 19502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:54.359704 19502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:54.359871 19502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:54.363806 19502 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:38227
I20260928 21:52:54.363860 19569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:38227 every 8 connection(s)
I20260928 21:52:54.364223 19502 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:52:54.365738 19570 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:52:54.366390 19570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Bootstrap starting.
I20260928 21:52:54.370904 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 19502
I20260928 21:52:54.371169 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:35837
--local_ip_for_outbound_sockets=127.3.0.193
--tserver_master_addrs=127.3.0.254:38227
--webserver_port=35741
--webserver_interface=127.3.0.193
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:52:54.375337 19570 log.cc:824] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Log is configured to *not* fsync() on all Append() calls
I20260928 21:52:54.378533 19570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:52:54.378887 19570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Bootstrap complete.
I20260928 21:52:54.382189 19570 raft_consensus.cc:348] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:54.382511 19570 raft_consensus.cc:729] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a347157b6f7b4abe8f144601793d40df, State: Initialized, Role: FOLLOWER
I20260928 21:52:54.382680 19570 consensus_queue.cc:260] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:54.382741 19570 raft_consensus.cc:388] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:52:54.382783 19570 raft_consensus.cc:482] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:52:54.382836 19570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:52:54.384161 19570 raft_consensus.cc:504] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:54.384275 19570 leader_election.cc:302] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a347157b6f7b4abe8f144601793d40df; no voters:
I20260928 21:52:54.384439 19570 leader_election.cc:288] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 21:52:54.384534 19574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:52:54.384716 19574 raft_consensus.cc:686] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [term 3 LEADER]: Becoming Leader. State: Replica: a347157b6f7b4abe8f144601793d40df, State: Running, Role: LEADER
I20260928 21:52:54.384791 19570 sys_catalog.cc:564] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:52:54.384864 19574 consensus_queue.cc:237] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } }
I20260928 21:52:54.385280 19575 sys_catalog.cc:455] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a347157b6f7b4abe8f144601793d40df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } } }
I20260928 21:52:54.385365 19575 sys_catalog.cc:458] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: This master's current role is: LEADER
I20260928 21:52:54.385336 19576 sys_catalog.cc:455] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: SysCatalogTable state changed. Reason: New leader a347157b6f7b4abe8f144601793d40df. Latest consensus state: current_term: 3 leader_uuid: "a347157b6f7b4abe8f144601793d40df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a347157b6f7b4abe8f144601793d40df" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38227 } } }
I20260928 21:52:54.385411 19576 sys_catalog.cc:458] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df [sys.catalog]: This master's current role is: LEADER
I20260928 21:52:54.385571 19580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:52:54.386400 19580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:52:54.386502 19580 catalog_manager.cc:1259] Loaded cluster ID: de27b1c2e9b24b6894d387e460a99b39
I20260928 21:52:54.386556 19580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:52:54.387557 19580 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:52:54.387724 19580 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Loaded TSK: 0
I20260928 21:52:54.387761 19580 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a347157b6f7b4abe8f144601793d40df: Loaded TSK: 1
F20260928 21:52:54.387972 19580 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb0f7a139ad google::LogMessage::Fail() at ??:0
@ 0x7fb0f7a1688a google::LogMessage::SendToLog() at ??:0
@ 0x7fb0f7a134de google::LogMessage::Flush() at ??:0
@ 0x7fb0f7a14a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb0fa180104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb0fa17dd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb0ff96d1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb0ff96d48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb0ff96d594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb0ff933c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb0ff93dad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb0f98e10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb0f98d6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb0fa73a6db start_thread at ??:0
@ 0x7fb0f651161f clone at ??:0
W20260928 21:52:54.469347 19572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:54.469554 19572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:54.469588 19572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:54.470752 19572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:54.470793 19572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:54.470829 19572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:52:54.472307 19572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:35837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=35741
--tserver_master_addrs=127.3.0.254:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:54.472573 19572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:54.472781 19572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:52:54.477909 19572 server_base.cc:1034] running on GCE node
W20260928 21:52:54.477968 19614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:54.478011 19611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:54.478242 19612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:54.478457 19572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:54.478696 19572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:54.479854 19572 hybrid_clock.cc:648] HybridClock initialized: now 1790632374479844 us; error 27 us; skew 500 ppm
Sep 28 21:52:54 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632374, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:54.482241 19572 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:52:54.482698 19572 webserver.cc:460] Webserver started at http://127.3.0.193:35741/ using document root <none> and password file <none>
I20260928 21:52:54.482928 19572 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:54.482978 19572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:54.484254 19572 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:54.484946 19621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:54.485101 19572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:54.485183 19572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "0b755691f36c4e689d712cb193612fcf"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "0c8e166ff84aef515d1899a6b606ac05"
server_key_iv: "759efaaffb76fb7aba219f3a69f18bbf"
server_key_version: "encryptionkey@0"
I20260928 21:52:54.485531 19572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:54.504802 19572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:54.506903 19572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:54.507038 19572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:54.507239 19572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:52:54.507647 19628 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:52:54.509624 19572 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:52:54.509675 19572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 21:52:54.509708 19572 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:52:54.510334 19572 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:52:54.510366 19572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 21:52:54.510459 19628 tablet_bootstrap.cc:492] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Bootstrap starting.
I20260928 21:52:54.517056 19572 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:35837
I20260928 21:52:54.517264 19735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:35837 every 8 connection(s)
I20260928 21:52:54.517493 19572 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:52:54.521179 19628 log.cc:824] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Log is configured to *not* fsync() on all Append() calls
I20260928 21:52:54.523943 19628 tablet_bootstrap.cc:492] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:52:54.524266 19628 tablet_bootstrap.cc:492] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Bootstrap complete.
I20260928 21:52:54.525421 19628 ts_tablet_manager.cc:1397] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260928 21:52:54.526995 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 19572
I20260928 21:52:54.526876 19628 raft_consensus.cc:348] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:54.527217 19628 raft_consensus.cc:729] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b755691f36c4e689d712cb193612fcf, State: Initialized, Role: FOLLOWER
I20260928 21:52:54.527247 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:37769
--local_ip_for_outbound_sockets=127.3.0.194
--tserver_master_addrs=127.3.0.254:38227
--webserver_port=39567
--webserver_interface=127.3.0.194
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:52:54.527401 19628 consensus_queue.cc:260] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:54.527697 19628 ts_tablet_manager.cc:1428] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260928 21:52:54.592780 19738 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:52:54.518114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:42739)
0928 21:52:54.518280 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:54.518323 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:54.518767 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:54.592574 (+ 73807us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
W20260928 21:52:54.593026 19736 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:38227 (0 consecutive failures): Network error: Failed to ping master at 127.3.0.254:38227: Client connection negotiation failed: client connection to 127.3.0.254:38227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:52:54.593358 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:54.593253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:58163)
0928 21:52:54.593298 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:54.593337 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 21:52:54.593653 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:54.593570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:43675)
0928 21:52:54.593610 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:54.593633 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:52:54.593760 19736 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:52:54.622424 19741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:54.622618 19741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:54.622641 19741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:54.623785 19741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:54.623828 19741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:54.623854 19741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:52:54.625332 19741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:37769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=39567
--tserver_master_addrs=127.3.0.254:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:54.625523 19741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:54.625700 19741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:52:54.630292 19747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:54.630358 19749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:54.630318 19746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:54.630487 19741 server_base.cc:1034] running on GCE node
I20260928 21:52:54.630714 19741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:54.630928 19741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:54.632079 19741 hybrid_clock.cc:648] HybridClock initialized: now 1790632374632066 us; error 27 us; skew 500 ppm
Sep 28 21:52:54 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632374, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:54.634485 19741 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:52:54.634865 19741 webserver.cc:460] Webserver started at http://127.3.0.194:39567/ using document root <none> and password file <none>
I20260928 21:52:54.635047 19741 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:54.635087 19741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:54.636334 19741 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:52:54.636919 19756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:54.637118 19741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:54.637202 19741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "ee397f845f8047979795c00290b51aee"
format_stamp: "Formatted at 2026-09-28 21:52:12 on dist-test-slave-jrck"
server_key: "3dc548adc42e580fbcc139c2044c0a65"
server_key_iv: "2a5903cccb4f36af0b329930cae36f24"
server_key_version: "encryptionkey@0"
I20260928 21:52:54.637531 19741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:54.662432 19741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:54.664381 19741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:54.664541 19741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:54.664779 19741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:52:54.665082 19741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:52:54.665115 19741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:54.665138 19741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:52:54.665160 19741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:54.670675 19741 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:37769
I20260928 21:52:54.670732 19869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:37769 every 8 connection(s)
I20260928 21:52:54.671059 19741 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:52:54.672066 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:54.671695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:54659)
0928 21:52:54.671843 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:54.671923 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260928 21:52:54.672503 19870 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:38227 (0 consecutive failures): Network error: Failed to ping master at 127.3.0.254:38227: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
I20260928 21:52:54.672753 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:54.672672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:41959)
0928 21:52:54.672709 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:54.672736 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 21:52:54.672979 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 19741
I20260928 21:52:54.673012 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:54.672935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:50363)
0928 21:52:54.672970 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:54.672995 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 21:52:54.673095 19870 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:52:54.673243 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:37381
--local_ip_for_outbound_sockets=127.3.0.195
--tserver_master_addrs=127.3.0.254:38227
--webserver_port=40587
--webserver_interface=127.3.0.195
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:34643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:52:54.753041 19873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:52:54.753257 19873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:52:54.753279 19873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:52:54.754447 19873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:52:54.754492 19873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:52:54.754518 19873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:52:54.756006 19873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:34643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:37381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=40587
--tserver_master_addrs=127.3.0.254:38227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:52:54.756230 19873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:52:54.756506 19873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:52:54.761461 19878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:54.761433 19881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:52:54.761447 19879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:52:54.761672 19873 server_base.cc:1034] running on GCE node
I20260928 21:52:54.761898 19873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:52:54.762138 19873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:52:54.763295 19873 hybrid_clock.cc:648] HybridClock initialized: now 1790632374763279 us; error 30 us; skew 500 ppm
Sep 28 21:52:54 dist-test-slave-jrck krb5kdc[18432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632374, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:52:54.765808 19873 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:52:54.766281 19873 webserver.cc:460] Webserver started at http://127.3.0.195:40587/ using document root <none> and password file <none>
I20260928 21:52:54.766517 19873 fs_manager.cc:359] Metadata directory not provided
I20260928 21:52:54.766569 19873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:52:54.767874 19873 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:54.768519 19888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:54.768669 19873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:52:54.768751 19873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "3c8f1afe27ff436e99f9cda9b87d2e40"
format_stamp: "Formatted at 2026-09-28 21:52:13 on dist-test-slave-jrck"
server_key: "7c54591c844e036061c2fc97476ac422"
server_key_iv: "1eb152d7c4ff077ba5807c5a6ce91919"
server_key_version: "encryptionkey@0"
I20260928 21:52:54.769091 19873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:52:54.780115 19873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:52:54.781996 19873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:52:54.782130 19873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:52:54.782351 19873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:52:54.782644 19873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:52:54.782675 19873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:54.782703 19873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:52:54.782727 19873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:52:54.788596 19873 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:37381
I20260928 21:52:54.788650 20001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:37381 every 8 connection(s)
I20260928 21:52:54.788982 19873 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:52:54.790082 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:54.789691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:35347)
0928 21:52:54.789872 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:54.789943 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
W20260928 21:52:54.790450 20002 heartbeater.cc:644] Failed to heartbeat to 127.3.0.254:38227 (0 consecutive failures): Network error: Failed to ping master at 127.3.0.254:38227: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
I20260928 21:52:54.790700 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:54.790632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:40937)
0928 21:52:54.790659 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:54.790684 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 21:52:54.790939 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:54.790864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:33703)
0928 21:52:54.790903 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:54.790924 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 21:52:54.791025 20002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:52:54.798102 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 19873
I20260928 21:52:55.594389 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:55.594135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:35665)
0928 21:52:55.594281 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:55.594327 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
I20260928 21:52:55.673640 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:55.673402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:45791)
0928 21:52:55.673555 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:55.673592 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":57,"threads_started":1}
I20260928 21:52:55.791517 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:55.791328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:41051)
0928 21:52:55.791432 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:55.791467 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260928 21:52:56.353575 20009 raft_consensus.cc:482] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:52:56.353700 20009 raft_consensus.cc:504] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:56.353988 20009 leader_election.cc:288] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381), ee397f845f8047979795c00290b51aee (127.3.0.194:37769)
Sep 28 21:52:56 dist-test-slave-jrck krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632374, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.194@KRBTEST.COM
Sep 28 21:52:56 dist-test-slave-jrck krb5kdc[18432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632374, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.195@KRBTEST.COM
I20260928 21:52:56.363782 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:56.354304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:37769 (local address 127.3.0.193:36581)
0928 21:52:56.354418 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:56.354427 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:56.354481 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:56.355412 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:56.355413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:56.355684 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:52:56.355904 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:56.355910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:56.357103 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:56.357104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:56.357885 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:56.357887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:56.358432 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:56.358434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:56.358542 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:56.359086 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:52:56.359090 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:52:56.360414 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:52:56.362228 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:56.362235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:56.362237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:56.362504 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:56.362716 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:56.362717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:56.362717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:56.362850 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:56.363246 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:52:56.363253 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:52:56.363408 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 21:52:56.363519 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:52:56.363670 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":37,"thread_start_us":60,"threads_started":1}
I20260928 21:52:56.363814 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:56.354442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:36581 (local address 127.3.0.194:37769)
0928 21:52:56.354578 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:56.354585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:56.354604 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:52:56.355074 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:56.355077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:56.355118 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:56.355286 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:56.356020 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:56.357014 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:56.357963 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:56.358346 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:56.358450 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:56.360540 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:56.360544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:56.360545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:56.360556 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:56.362149 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:56.362580 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:56.362582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:56.362582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:56.362658 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:56.362907 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:56.362908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:56.362909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:56.363178 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:56.363258 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:56.363576 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 21:52:56.363706 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260928 21:52:56.363905 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:56.354163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:37381 (local address 127.3.0.193:36853)
0928 21:52:56.354330 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:56.354342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:52:56.354416 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:52:56.355334 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:52:56.355336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:52:56.355688 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:52:56.355904 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:56.355910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:56.357083 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:56.357084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:56.357884 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:52:56.357886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:52:56.358427 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:52:56.358428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:52:56.358542 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:56.359042 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:52:56.359049 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:52:56.360816 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:52:56.362717 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:56.362718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:56.362719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:56.362931 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:56.363138 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:52:56.363139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:52:56.363139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:52:56.363197 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:52:56.363617 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:52:56.363618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:52:56.363659 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 21:52:56.363735 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 21:52:56.363823 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":52,"thread_start_us":56,"threads_started":1}
I20260928 21:52:56.364261 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:56.354263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:36853 (local address 127.3.0.195:37381)
0928 21:52:56.354422 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:52:56.354435 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:52:56.354463 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:52:56.355053 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:52:56.355057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:52:56.355118 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:52:56.355286 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:52:56.356028 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:56.357000 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:56.357966 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:52:56.358346 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:52:56.358439 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:52:56.360932 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:52:56.360936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:52:56.360936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:52:56.360948 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:52:56.362649 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:56.363005 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:56.363008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:56.363008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:56.363085 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:52:56.363281 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:52:56.363282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:52:56.363282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:52:56.363549 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:52:56.363619 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:52:56.363802 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 21:52:56.364153 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":53,"threads_started":1}
W20260928 21:52:56.364748 19622 leader_election.cc:341] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee397f845f8047979795c00290b51aee (127.3.0.194:37769): Not found: Tablet not found: f1db8ad904b444eb8832e20f7a8fb7da
W20260928 21:52:56.364850 19622 leader_election.cc:341] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381): Not found: Tablet not found: f1db8ad904b444eb8832e20f7a8fb7da
I20260928 21:52:56.364900 19622 leader_election.cc:302] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b755691f36c4e689d712cb193612fcf; no voters: 3c8f1afe27ff436e99f9cda9b87d2e40, ee397f845f8047979795c00290b51aee
I20260928 21:52:56.364990 20009 raft_consensus.cc:2726] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:52:56.594964 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:56.594814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:39091)
0928 21:52:56.594891 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:56.594922 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:52:56.674299 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:56.674077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:57709)
0928 21:52:56.674207 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:56.674256 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260928 21:52:56.792156 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:56.791946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:49525)
0928 21:52:56.792066 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:56.792112 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260928 21:52:57.595585 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:57.595346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:59435)
0928 21:52:57.595497 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:57.595537 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260928 21:52:57.674948 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:57.674735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:58065)
0928 21:52:57.674858 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:57.674903 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260928 21:52:57.792846 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:57.792632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:47835)
0928 21:52:57.792755 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:57.792800 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260928 21:52:58.347529 20019 raft_consensus.cc:482] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:52:58.347630 20019 raft_consensus.cc:504] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:58.347787 20019 leader_election.cc:288] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381), ee397f845f8047979795c00290b51aee (127.3.0.194:37769)
W20260928 21:52:58.348202 19622 leader_election.cc:341] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee397f845f8047979795c00290b51aee (127.3.0.194:37769): Not found: Tablet not found: f1db8ad904b444eb8832e20f7a8fb7da
W20260928 21:52:58.348276 19622 leader_election.cc:341] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381): Not found: Tablet not found: f1db8ad904b444eb8832e20f7a8fb7da
I20260928 21:52:58.348306 19622 leader_election.cc:302] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b755691f36c4e689d712cb193612fcf; no voters: 3c8f1afe27ff436e99f9cda9b87d2e40, ee397f845f8047979795c00290b51aee
I20260928 21:52:58.348390 20019 raft_consensus.cc:2726] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:52:58.596228 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:58.596003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:37935)
0928 21:52:58.596147 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:58.596180 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260928 21:52:58.675585 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:58.675378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:43491)
0928 21:52:58.675495 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:58.675533 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260928 21:52:58.793500 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:58.793273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:54555)
0928 21:52:58.793408 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:58.793446 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260928 21:52:59.596918 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:59.596698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:34325)
0928 21:52:59.596834 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:59.596870 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260928 21:52:59.676235 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:59.676044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:47401)
0928 21:52:59.676159 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:59.676195 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260928 21:52:59.794121 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:52:59.793931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:48837)
0928 21:52:59.794039 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:52:59.794076 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260928 21:52:59.925280 20026 raft_consensus.cc:482] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:52:59.925379 20026 raft_consensus.cc:504] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:52:59.925504 20026 leader_election.cc:288] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381), ee397f845f8047979795c00290b51aee (127.3.0.194:37769)
W20260928 21:52:59.925868 19622 leader_election.cc:341] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381): Not found: Tablet not found: f1db8ad904b444eb8832e20f7a8fb7da
W20260928 21:52:59.925946 19622 leader_election.cc:341] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee397f845f8047979795c00290b51aee (127.3.0.194:37769): Not found: Tablet not found: f1db8ad904b444eb8832e20f7a8fb7da
I20260928 21:52:59.925968 19622 leader_election.cc:302] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b755691f36c4e689d712cb193612fcf; no voters: 3c8f1afe27ff436e99f9cda9b87d2e40, ee397f845f8047979795c00290b51aee
I20260928 21:52:59.926034 20026 raft_consensus.cc:2726] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:53:00.597605 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:00.597400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:46705)
0928 21:53:00.597519 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:00.597558 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260928 21:53:00.676842 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:00.676651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:33111)
0928 21:53:00.676762 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:00.676798 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260928 21:53:00.794715 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:00.794505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:52283)
0928 21:53:00.794631 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:00.794672 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260928 21:53:01.598228 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:01.598007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:49061)
0928 21:53:01.598143 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:01.598180 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260928 21:53:01.677498 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:01.677280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:37849)
0928 21:53:01.677405 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:01.677448 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260928 21:53:01.679690 20032 raft_consensus.cc:482] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:53:01.679792 20032 raft_consensus.cc:504] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:53:01.679922 20032 leader_election.cc:288] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381), ee397f845f8047979795c00290b51aee (127.3.0.194:37769)
W20260928 21:53:01.680357 19622 leader_election.cc:341] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381): Not found: Tablet not found: f1db8ad904b444eb8832e20f7a8fb7da
W20260928 21:53:01.680440 19622 leader_election.cc:341] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee397f845f8047979795c00290b51aee (127.3.0.194:37769): Not found: Tablet not found: f1db8ad904b444eb8832e20f7a8fb7da
I20260928 21:53:01.680502 19622 leader_election.cc:302] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b755691f36c4e689d712cb193612fcf; no voters: 3c8f1afe27ff436e99f9cda9b87d2e40, ee397f845f8047979795c00290b51aee
I20260928 21:53:01.680589 20032 raft_consensus.cc:2726] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:53:01.795337 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:01.795133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:56997)
0928 21:53:01.795257 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:01.795292 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260928 21:53:02.598892 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:02.598655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:35513)
0928 21:53:02.598805 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:02.598846 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260928 21:53:02.678073 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:02.677879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:53897)
0928 21:53:02.677992 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:02.678027 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260928 21:53:02.795985 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:02.795774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:39463)
0928 21:53:02.795903 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:02.795941 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260928 21:53:03.599575 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:03.599362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:44451)
0928 21:53:03.599481 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:03.599525 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260928 21:53:03.610282 20038 raft_consensus.cc:482] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:53:03.610369 20038 raft_consensus.cc:504] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c8f1afe27ff436e99f9cda9b87d2e40" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 37381 } } peers { permanent_uuid: "ee397f845f8047979795c00290b51aee" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37769 } } peers { permanent_uuid: "0b755691f36c4e689d712cb193612fcf" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 35837 } }
I20260928 21:53:03.610518 20038 leader_election.cc:288] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381), ee397f845f8047979795c00290b51aee (127.3.0.194:37769)
W20260928 21:53:03.610911 19622 leader_election.cc:341] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee397f845f8047979795c00290b51aee (127.3.0.194:37769): Not found: Tablet not found: f1db8ad904b444eb8832e20f7a8fb7da
W20260928 21:53:03.610984 19622 leader_election.cc:341] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c8f1afe27ff436e99f9cda9b87d2e40 (127.3.0.195:37381): Not found: Tablet not found: f1db8ad904b444eb8832e20f7a8fb7da
I20260928 21:53:03.611017 19622 leader_election.cc:302] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b755691f36c4e689d712cb193612fcf; no voters: 3c8f1afe27ff436e99f9cda9b87d2e40, ee397f845f8047979795c00290b51aee
I20260928 21:53:03.611099 20038 raft_consensus.cc:2726] T f1db8ad904b444eb8832e20f7a8fb7da P 0b755691f36c4e689d712cb193612fcf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:53:03.678732 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:03.678492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:51525)
0928 21:53:03.678621 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:03.678687 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260928 21:53:03.796653 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:03.796374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:46775)
0928 21:53:03.796542 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:03.796601 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260928 21:53:04.600210 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:04.599997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:60231)
0928 21:53:04.600120 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:04.600160 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260928 21:53:04.679381 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:04.679154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:43613)
0928 21:53:04.679296 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:04.679334 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260928 21:53:04.797428 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:04.797164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:33821)
0928 21:53:04.797336 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:04.797373 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260928 21:53:05.600973 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:05.600735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:52121)
0928 21:53:05.600879 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:05.600925 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260928 21:53:05.680130 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:05.679893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:35343)
0928 21:53:05.680032 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:05.680080 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260928 21:53:05.798131 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:05.797850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:46545)
0928 21:53:05.798009 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:05.798081 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260928 21:53:06.601775 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:06.601500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:36057)
0928 21:53:06.601681 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:06.601725 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":65,"threads_started":1}
I20260928 21:53:06.680940 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:06.680684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:59939)
0928 21:53:06.680837 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:06.680888 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260928 21:53:06.798945 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:06.798691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:54287)
0928 21:53:06.798845 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:06.798895 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260928 21:53:07.602658 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:07.602384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:41967)
0928 21:53:07.602538 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:07.602589 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260928 21:53:07.681735 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:07.681502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:47593)
0928 21:53:07.681643 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:07.681687 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260928 21:53:07.799738 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:07.799520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:51985)
0928 21:53:07.799646 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:07.799688 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260928 21:53:08.603365 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:08.603141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:55823)
0928 21:53:08.603273 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:08.603314 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260928 21:53:08.682440 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:08.682201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:60171)
0928 21:53:08.682348 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:08.682392 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260928 21:53:08.800412 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:08.800165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:43755)
0928 21:53:08.800322 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:08.800365 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260928 21:53:09.604218 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:09.603973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.193:36031)
0928 21:53:09.604123 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:09.604171 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260928 21:53:09.683259 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:09.683008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.194:52321)
0928 21:53:09.683161 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:09.683210 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
I20260928 21:53:09.801237 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:09.801003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38227 (local address 127.3.0.195:52325)
0928 21:53:09.801143 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:09.801190 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.254:38227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260928 21:53:09.804180 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 19572
I20260928 21:53:09.808430 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 19741
I20260928 21:53:09.812318 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 19873
2026-09-28T21:53:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58391 ms)
[----------] 30 tests from SecurityITest (250468 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:09 dist-test-slave-jrck krb5kdc[20061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:09 dist-test-slave-jrck krb5kdc[20061](info): set up 2 sockets
Sep 28 21:53:09 dist-test-slave-jrck krb5kdc[20061](info): commencing operation
krb5kdc: starting...
W20260928 21:53:10.776221 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:10 dist-test-slave-jrck krb5kdc[20061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:10.887959 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:44387
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:10.967106 20074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:10.967338 20074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:10.967361 20074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:10.968573 20074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:10.968619 20074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:10.968632 20074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:10.968644 20074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:10.968655 20074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:10.970638 20074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:10.970908 20074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:10.971180 20074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:10.976868 20079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:10.977175 20074 server_base.cc:1034] running on GCE node
W20260928 21:53:10.976861 20082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:10.976943 20080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:10.977547 20074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:10.977900 20074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:10.979054 20074 hybrid_clock.cc:648] HybridClock initialized: now 1790632390979041 us; error 39 us; skew 500 ppm
Sep 28 21:53:10 dist-test-slave-jrck krb5kdc[20061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632390, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:10.981482 20074 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:10.981878 20074 webserver.cc:460] Webserver started at http://127.3.0.254:34275/ using document root <none> and password file <none>
I20260928 21:53:10.982105 20074 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:10.982154 20074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:10.982275 20074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:10.983378 20074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "9a5f9f43aa0542f3af310b7e512c5104"
format_stamp: "Formatted at 2026-09-28 21:53:10 on dist-test-slave-jrck"
server_key: "31d4dea771c713a78ef6ba3a5ff65f39"
server_key_iv: "c7f8fe3bb7571d585abfa9312739b401"
server_key_version: "encryptionkey@0"
I20260928 21:53:10.983693 20074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "9a5f9f43aa0542f3af310b7e512c5104"
format_stamp: "Formatted at 2026-09-28 21:53:10 on dist-test-slave-jrck"
server_key: "31d4dea771c713a78ef6ba3a5ff65f39"
server_key_iv: "c7f8fe3bb7571d585abfa9312739b401"
server_key_version: "encryptionkey@0"
I20260928 21:53:10.984994 20074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 21:53:10.985723 20089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:10.985886 20074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:10.985980 20074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "9a5f9f43aa0542f3af310b7e512c5104"
format_stamp: "Formatted at 2026-09-28 21:53:10 on dist-test-slave-jrck"
server_key: "31d4dea771c713a78ef6ba3a5ff65f39"
server_key_iv: "c7f8fe3bb7571d585abfa9312739b401"
server_key_version: "encryptionkey@0"
I20260928 21:53:10.986049 20074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:10.998436 20074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:11.000615 20074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:11.000753 20074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:11.004570 20074 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:44387
I20260928 21:53:11.004607 20141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:44387 every 8 connection(s)
I20260928 21:53:11.004936 20074 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:11.005770 20142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:11.008075 20142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104: Bootstrap starting.
I20260928 21:53:11.008674 20142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:11.008951 20142 log.cc:824] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:11.009685 20142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104: No bootstrap required, opened a new log
I20260928 21:53:11.011301 20142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f9f43aa0542f3af310b7e512c5104" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44387 } }
I20260928 21:53:11.011441 20142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:11.011473 20142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5f9f43aa0542f3af310b7e512c5104, State: Initialized, Role: FOLLOWER
I20260928 21:53:11.011617 20142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f9f43aa0542f3af310b7e512c5104" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44387 } }
I20260928 21:53:11.011680 20142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:11.011703 20142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:11.011746 20142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:11.012255 20142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f9f43aa0542f3af310b7e512c5104" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44387 } }
I20260928 21:53:11.012344 20142 leader_election.cc:302] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a5f9f43aa0542f3af310b7e512c5104; no voters:
I20260928 21:53:11.012537 20142 leader_election.cc:288] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:11.012563 20145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:11.012617 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 20074
I20260928 21:53:11.012719 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:11.012727 20145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [term 1 LEADER]: Becoming Leader. State: Replica: 9a5f9f43aa0542f3af310b7e512c5104, State: Running, Role: LEADER
I20260928 21:53:11.012805 20142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:11.012807 20145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f9f43aa0542f3af310b7e512c5104" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44387 } }
I20260928 21:53:11.012890 3075 external_mini_cluster.cc:1428] Setting key 1bfef48d5bed398da4dc901075dc7513
I20260928 21:53:11.013163 20145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a5f9f43aa0542f3af310b7e512c5104. Latest consensus state: current_term: 1 leader_uuid: "9a5f9f43aa0542f3af310b7e512c5104" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f9f43aa0542f3af310b7e512c5104" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44387 } } }
I20260928 21:53:11.013648 20145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:11.013863 20146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a5f9f43aa0542f3af310b7e512c5104" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5f9f43aa0542f3af310b7e512c5104" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44387 } } }
I20260928 21:53:11.013985 20146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:11.014118 20158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:11.014721 20158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:11.016212 20158 catalog_manager.cc:1347] Generated new cluster ID: 2729000512d14c25a0c9b83dd98efed2
I20260928 21:53:11.016264 20158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:53:11 dist-test-slave-jrck krb5kdc[20061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:11.024745 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.013511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.3.0.254:44387)
0928 21:53:11.013797 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.013806 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.013840 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:53:11.013956 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.013964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.014055 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.014261 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:11.014968 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.016135 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.017245 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.017846 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.017939 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.020585 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:11.020589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:11.020589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:11.020601 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:11.023063 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:11.023450 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:11.023453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:11.023453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:11.023591 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:11.023764 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:11.023765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:11.023765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:11.024090 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:11.024245 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.024319 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.024492 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":74,"threads_started":1}
I20260928 21:53:11.027098 20158 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:11.027578 20158 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:11.032155 20158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104: Generated new TSK 0
I20260928 21:53:11.032326 20158 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:53:11.063122 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:44387
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:11.143616 20168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:11.143838 20168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:11.143864 20168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:11.145071 20168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:11.145115 20168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:11.145141 20168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:53:11.146618 20168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:11.146850 20168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:11.147075 20168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:11.151773 20174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:11.151767 20176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:11.151767 20173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:11.152148 20168 server_base.cc:1034] running on GCE node
I20260928 21:53:11.152341 20168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:11.152611 20168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:11.153744 20168 hybrid_clock.cc:648] HybridClock initialized: now 1790632391153731 us; error 30 us; skew 500 ppm
Sep 28 21:53:11 dist-test-slave-jrck krb5kdc[20061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:11.156065 20168 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:53:11.156543 20168 webserver.cc:460] Webserver started at http://127.3.0.193:44113/ using document root <none> and password file <none>
I20260928 21:53:11.156752 20168 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:11.156800 20168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:11.156925 20168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:11.157994 20168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "a4770db0c5614bbaae7128e40b32300b"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "9ee93448a7c5f283690eefad1a367348"
server_key_iv: "8134a5741d273fa5a5d6e9cdd0354ec9"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.158326 20168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "a4770db0c5614bbaae7128e40b32300b"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "9ee93448a7c5f283690eefad1a367348"
server_key_iv: "8134a5741d273fa5a5d6e9cdd0354ec9"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.159572 20168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:11.160310 20183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.160528 20168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:53:11.160615 20168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "a4770db0c5614bbaae7128e40b32300b"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "9ee93448a7c5f283690eefad1a367348"
server_key_iv: "8134a5741d273fa5a5d6e9cdd0354ec9"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.160686 20168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:11.175288 20168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:11.177333 20168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:11.177471 20168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:11.177704 20168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:11.177999 20168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:11.178033 20168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.178062 20168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:11.178092 20168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.183460 20168 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:44825
I20260928 21:53:11.183511 20296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:44825 every 8 connection(s)
I20260928 21:53:11.183840 20168 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:53:11.188370 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 20168
I20260928 21:53:11.188516 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:53:11.188721 3075 external_mini_cluster.cc:1428] Setting key b4c31e628defd8a94324c587301c5962
Sep 28 21:53:11 dist-test-slave-jrck krb5kdc[20061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:11.193341 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.184529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:57643 (local address 127.3.0.254:44387)
0928 21:53:11.184586 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.184588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.184777 (+ 189us) server_negotiation.cc:407] Connection header received
0928 21:53:11.185292 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.185293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.185307 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.185360 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:11.186351 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.186875 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.187869 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.188085 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.188138 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.190481 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:11.190483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:11.190484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:11.190492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:11.191883 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:11.192344 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:11.192345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:11.192345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:11.192398 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:11.192698 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:11.192699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:11.192699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:11.192809 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:11.192856 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.193196 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.193273 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:53:11.193639 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.184507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44387 (local address 127.3.0.193:57643)
0928 21:53:11.184657 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:11.184695 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:11.185189 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:11.185435 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:11.185440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:11.186042 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:11.186247 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.186252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.186971 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.186972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.187782 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.187784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.188179 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.188180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.188307 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.188883 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:11.188892 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:11.190355 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:11.191963 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:11.191972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:11.191974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:11.192265 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:11.192482 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:11.192483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:11.192483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:11.192633 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:11.192863 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:11.192869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:11.193052 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 21:53:11.193166 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:53:11.193349 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260928 21:53:11.194211 20297 heartbeater.cc:347] Connected to a master server at 127.3.0.254:44387
I20260928 21:53:11.194319 20297 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:11.194532 20297 heartbeater.cc:511] Master 127.3.0.254:44387 requested a full tablet report, sending...
I20260928 21:53:11.195076 20106 ts_manager.cc:194] Registered new tserver with Master: a4770db0c5614bbaae7128e40b32300b (127.3.0.193:44825)
I20260928 21:53:11.195664 20106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:57643
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:53:11.215070 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:44387
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:11.294528 20302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:11.294731 20302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:11.294762 20302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:11.295907 20302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:11.295946 20302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:11.295979 20302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:53:11.297448 20302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:11.297680 20302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:11.297887 20302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:11.302552 20307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:11.302562 20308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:11.302758 20302 server_base.cc:1034] running on GCE node
W20260928 21:53:11.302562 20310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:11.303052 20302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:11.303283 20302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:11.304441 20302 hybrid_clock.cc:648] HybridClock initialized: now 1790632391304429 us; error 30 us; skew 500 ppm
Sep 28 21:53:11 dist-test-slave-jrck krb5kdc[20061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:11.306846 20302 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:53:11.307258 20302 webserver.cc:460] Webserver started at http://127.3.0.194:37783/ using document root <none> and password file <none>
I20260928 21:53:11.307463 20302 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:11.307518 20302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:11.307641 20302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:11.308717 20302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "f33e2437df314c358beeea8e516a82d6"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "cf9f19a4d8339c7e1530a6497cb52058"
server_key_iv: "e5b4261097a08ca6da3275e482029111"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.309041 20302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "f33e2437df314c358beeea8e516a82d6"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "cf9f19a4d8339c7e1530a6497cb52058"
server_key_iv: "e5b4261097a08ca6da3275e482029111"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.310263 20302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 21:53:11.310971 20317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.311143 20302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.311223 20302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "f33e2437df314c358beeea8e516a82d6"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "cf9f19a4d8339c7e1530a6497cb52058"
server_key_iv: "e5b4261097a08ca6da3275e482029111"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.311280 20302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:11.322494 20302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:11.324638 20302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:11.324779 20302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:11.324991 20302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:11.325289 20302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:11.325323 20302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.325352 20302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:11.325376 20302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.330972 20302 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:39141
I20260928 21:53:11.331040 20430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:39141 every 8 connection(s)
I20260928 21:53:11.331342 20302 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:53:11 dist-test-slave-jrck krb5kdc[20061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:11.339890 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 20302
I20260928 21:53:11.340063 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:53:11.340260 3075 external_mini_cluster.cc:1428] Setting key e5b5338ef219b6543f1a8c63569f0a72
I20260928 21:53:11.341076 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.332065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:50177 (local address 127.3.0.254:44387)
0928 21:53:11.332159 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.332162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.332336 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:53:11.333003 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.333004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.333017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.333063 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:11.334065 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.334572 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.335626 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.335812 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.335868 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.338121 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:11.338123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:11.338123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:11.338131 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:11.339487 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:11.339918 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:11.339919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:11.339920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:11.339992 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:11.340266 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:11.340267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:11.340268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:11.340443 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:11.340533 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.340854 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.340954 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:53:11.341285 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.331997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44387 (local address 127.3.0.194:50177)
0928 21:53:11.332201 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:11.332249 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:11.332894 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:11.333154 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:11.333159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:11.333738 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:11.333962 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.333967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.334661 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.334662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.335535 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.335538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.335890 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.335891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.336016 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.336551 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:11.336560 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:11.338002 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:11.339579 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:11.339587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:11.339589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:11.339820 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:11.340084 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:11.340084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:11.340085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:11.340203 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:11.340524 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:11.340532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:11.340714 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 21:53:11.340833 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:53:11.341034 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260928 21:53:11.341714 20431 heartbeater.cc:347] Connected to a master server at 127.3.0.254:44387
I20260928 21:53:11.341825 20431 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:11.342026 20431 heartbeater.cc:511] Master 127.3.0.254:44387 requested a full tablet report, sending...
I20260928 21:53:11.342435 20106 ts_manager.cc:194] Registered new tserver with Master: f33e2437df314c358beeea8e516a82d6 (127.3.0.194:39141)
I20260928 21:53:11.342798 20106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:50177
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:53:11.364130 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:44387
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:11.444959 20436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:11.445184 20436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:11.445218 20436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:11.446378 20436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:11.446421 20436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:11.446455 20436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:53:11.447948 20436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:44387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:11.448191 20436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:11.448426 20436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:11.454315 20444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:11.454315 20442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:11.454461 20436 server_base.cc:1034] running on GCE node
W20260928 21:53:11.454315 20441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:11.454697 20436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:11.454957 20436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:11.456100 20436 hybrid_clock.cc:648] HybridClock initialized: now 1790632391456086 us; error 29 us; skew 500 ppm
Sep 28 21:53:11 dist-test-slave-jrck krb5kdc[20061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:11.458645 20436 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:53:11.459103 20436 webserver.cc:460] Webserver started at http://127.3.0.195:41827/ using document root <none> and password file <none>
I20260928 21:53:11.459317 20436 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:11.459367 20436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:11.459501 20436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:11.460582 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "ded914ade3a244c88ba5cd5ef1c2c13e"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "23ea4ffe38fbf24ce132c3090297bb3e"
server_key_iv: "f62bad4dfda08d1a80156b249bc7b7db"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.460911 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "ded914ade3a244c88ba5cd5ef1c2c13e"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "23ea4ffe38fbf24ce132c3090297bb3e"
server_key_iv: "f62bad4dfda08d1a80156b249bc7b7db"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.462265 20436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:11.463016 20451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.463176 20436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:11.463264 20436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "ded914ade3a244c88ba5cd5ef1c2c13e"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "23ea4ffe38fbf24ce132c3090297bb3e"
server_key_iv: "f62bad4dfda08d1a80156b249bc7b7db"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.463317 20436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:11.470857 20436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:11.472994 20436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:11.473124 20436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:11.473308 20436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:11.473559 20436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:11.473590 20436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.473610 20436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:11.473626 20436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.479043 20436 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:38139
I20260928 21:53:11.479110 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:38139 every 8 connection(s)
I20260928 21:53:11.479395 20436 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:53:11 dist-test-slave-jrck krb5kdc[20061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632391, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:11.488284 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.480133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:52053 (local address 127.3.0.254:44387)
0928 21:53:11.480193 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.480195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.480375 (+ 180us) server_negotiation.cc:407] Connection header received
0928 21:53:11.480964 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.480966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.480978 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.481030 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:11.481987 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.482479 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.483483 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.483669 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.483722 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.485763 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:11.485765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:11.485765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:11.485773 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:11.487057 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:11.487434 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:11.487435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:11.487435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:11.487490 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:11.487700 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:11.487700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:11.487701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:11.487809 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:11.487862 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.488115 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.488192 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:53:11.488481 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.480088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44387 (local address 127.3.0.195:52053)
0928 21:53:11.480251 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:11.480291 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:11.480838 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:11.481106 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:11.481111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:11.481690 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:11.481893 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.481898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.482559 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.482560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.483400 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.483402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.483735 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.483736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.483851 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.484316 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:11.484325 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:11.485678 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:11.487143 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:11.487150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:11.487152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:11.487366 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:11.487545 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:11.487546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:11.487547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:11.487644 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:11.487861 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:11.487868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:11.487997 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 21:53:11.488085 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:53:11.488243 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260928 21:53:11.488907 20565 heartbeater.cc:347] Connected to a master server at 127.3.0.254:44387
I20260928 21:53:11.488977 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 20436
I20260928 21:53:11.489032 20565 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:11.489086 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:53:11.489217 20565 heartbeater.cc:511] Master 127.3.0.254:44387 requested a full tablet report, sending...
I20260928 21:53:11.489272 3075 external_mini_cluster.cc:1428] Setting key 09c065d412d1d866cb18e92328bd9114
I20260928 21:53:11.489657 20105 ts_manager.cc:194] Registered new tserver with Master: ded914ade3a244c88ba5cd5ef1c2c13e (127.3.0.195:38139)
I20260928 21:53:11.490079 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:53:11.490123 20105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:52053
I20260928 21:53:11.496791 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.491367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.3.0.254:44387)
0928 21:53:11.491406 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.491408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.491534 (+ 126us) server_negotiation.cc:407] Connection header received
0928 21:53:11.491594 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.491595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.491604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.491648 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:11.492167 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.492681 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.493398 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.493578 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.493630 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.494712 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:11.494713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:11.494714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:11.494720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:11.495738 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:11.496035 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:11.496036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:11.496037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:11.496086 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:11.496242 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:11.496242 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:11.496243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:11.496384 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:11.496496 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.496627 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.496703 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:53:11.498294 20105 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:53:11.499826 20105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:53:11.505998 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.502305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.3.0.195:38139)
0928 21:53:11.502514 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.502519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.502529 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:53:11.502586 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.502588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.502602 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.502670 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:11.502843 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.503923 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.505125 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.505615 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.505678 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.505737 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.505873 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.505912 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
I20260928 21:53:11.505988 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.502063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:38139 (local address 127.0.0.1:36026)
0928 21:53:11.502345 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:11.502353 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:11.502386 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:11.502739 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:11.502740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:11.502742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:11.502774 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.502775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.504039 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.504040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.505044 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.505047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.505693 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.505694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.505775 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.505789 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 21:53:11.505853 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:53:11.505909 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":34,"thread_start_us":73,"threads_started":1}
I20260928 21:53:11.506073 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.501958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.3.0.193:44825)
0928 21:53:11.502123 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.502128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.502302 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:53:11.502516 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.502518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.502533 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.502596 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:11.502821 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.503535 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.504905 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.505613 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.505679 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.505737 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.505944 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.505990 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260928 21:53:11.506326 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.502139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.3.0.194:39141)
0928 21:53:11.502281 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.502285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.502296 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:53:11.502392 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.502395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.502412 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.502505 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:11.502760 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.503932 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.505163 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.505827 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.505886 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.505941 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.506204 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.506248 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260928 21:53:11.506482 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.501858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39141 (local address 127.0.0.1:36462)
0928 21:53:11.502192 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:11.502235 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:11.502301 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:11.502606 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:11.502609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:11.502612 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:11.502682 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.502685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.504033 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.504034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.505100 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.505102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.505971 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.505972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.506023 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.506026 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:11.506069 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:53:11.506105 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":28,"thread_start_us":68,"threads_started":1}
I20260928 21:53:11.506577 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.501858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:44825 (local address 127.0.0.1:44428)
0928 21:53:11.502192 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:11.502235 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:11.502431 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:11.502683 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:11.502684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:11.502686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:11.502739 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.502740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.503632 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.503632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.504776 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.504779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.505770 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.505771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.505831 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.505835 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:53:11.506155 (+ 320us) client_negotiation.cc:240] Negotiation successful
0928 21:53:11.506190 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260928 21:53:11.507027 20499 tablet_service.cc:1467] Processing CreateTablet for tablet 2aae79459b684cc08662596cb8d8bbc2 (DEFAULT_TABLE table=test-table [id=f6a12fe0197c471cabdaec72bfca7ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:11.507074 20365 tablet_service.cc:1467] Processing CreateTablet for tablet 2aae79459b684cc08662596cb8d8bbc2 (DEFAULT_TABLE table=test-table [id=f6a12fe0197c471cabdaec72bfca7ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:11.507400 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aae79459b684cc08662596cb8d8bbc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:11.507400 20365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aae79459b684cc08662596cb8d8bbc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:11.509119 20231 tablet_service.cc:1467] Processing CreateTablet for tablet 2aae79459b684cc08662596cb8d8bbc2 (DEFAULT_TABLE table=test-table [id=f6a12fe0197c471cabdaec72bfca7ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:11.509428 20231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aae79459b684cc08662596cb8d8bbc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:11.510744 20580 tablet_bootstrap.cc:492] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: Bootstrap starting.
I20260928 21:53:11.510985 20579 tablet_bootstrap.cc:492] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: Bootstrap starting.
I20260928 21:53:11.511680 20579 tablet_bootstrap.cc:654] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:11.512231 20579 log.cc:824] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:11.512271 20581 tablet_bootstrap.cc:492] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b: Bootstrap starting.
I20260928 21:53:11.512960 20581 tablet_bootstrap.cc:654] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:11.513018 20579 tablet_bootstrap.cc:492] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: No bootstrap required, opened a new log
I20260928 21:53:11.513070 20580 tablet_bootstrap.cc:654] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:11.513082 20579 ts_tablet_manager.cc:1397] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:53:11.513517 20580 log.cc:824] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:11.513513 20581 log.cc:824] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:11.514253 20580 tablet_bootstrap.cc:492] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: No bootstrap required, opened a new log
I20260928 21:53:11.514259 20581 tablet_bootstrap.cc:492] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b: No bootstrap required, opened a new log
I20260928 21:53:11.514315 20581 ts_tablet_manager.cc:1397] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:53:11.514467 20580 ts_tablet_manager.cc:1397] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:53:11.514932 20579 raft_consensus.cc:348] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 } } peers { permanent_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 38139 } } peers { permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 } }
I20260928 21:53:11.515103 20579 raft_consensus.cc:374] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:11.515144 20579 raft_consensus.cc:729] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded914ade3a244c88ba5cd5ef1c2c13e, State: Initialized, Role: FOLLOWER
I20260928 21:53:11.515276 20579 consensus_queue.cc:260] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 } } peers { permanent_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 38139 } } peers { permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 } }
I20260928 21:53:11.515537 20579 ts_tablet_manager.cc:1428] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:53:11.515534 20565 heartbeater.cc:503] Master 127.3.0.254:44387 was elected leader, sending a full tablet report...
I20260928 21:53:11.516196 20581 raft_consensus.cc:348] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 } } peers { permanent_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 38139 } } peers { permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 } }
I20260928 21:53:11.516366 20581 raft_consensus.cc:374] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:11.516409 20581 raft_consensus.cc:729] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4770db0c5614bbaae7128e40b32300b, State: Initialized, Role: FOLLOWER
I20260928 21:53:11.516549 20581 consensus_queue.cc:260] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 } } peers { permanent_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 38139 } } peers { permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 } }
I20260928 21:53:11.516814 20581 ts_tablet_manager.cc:1428] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:53:11.516839 20297 heartbeater.cc:503] Master 127.3.0.254:44387 was elected leader, sending a full tablet report...
I20260928 21:53:11.516980 20580 raft_consensus.cc:348] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 } } peers { permanent_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 38139 } } peers { permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 } }
I20260928 21:53:11.517123 20580 raft_consensus.cc:374] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:11.517162 20580 raft_consensus.cc:729] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33e2437df314c358beeea8e516a82d6, State: Initialized, Role: FOLLOWER
I20260928 21:53:11.517268 20580 consensus_queue.cc:260] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 } } peers { permanent_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 38139 } } peers { permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 } }
I20260928 21:53:11.517532 20580 ts_tablet_manager.cc:1428] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:53:11.517535 20431 heartbeater.cc:503] Master 127.3.0.254:44387 was elected leader, sending a full tablet report...
I20260928 21:53:11.548666 20585 raft_consensus.cc:482] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:53:11.548820 20585 raft_consensus.cc:504] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 } } peers { permanent_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 38139 } } peers { permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 } }
I20260928 21:53:11.549264 20585 leader_election.cc:288] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4770db0c5614bbaae7128e40b32300b (127.3.0.193:44825), f33e2437df314c358beeea8e516a82d6 (127.3.0.194:39141)
I20260928 21:53:11.553061 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.549589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:53179 (local address 127.3.0.194:39141)
0928 21:53:11.549628 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.549629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.549707 (+ 78us) server_negotiation.cc:407] Connection header received
0928 21:53:11.549756 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.549757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.549767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.549807 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:11.549972 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.550696 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.552007 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.552626 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.552696 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.552706 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.552932 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.552985 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:53:11.553061 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.549534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:54613 (local address 127.3.0.193:44825)
0928 21:53:11.549584 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.549586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.549596 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:53:11.549626 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.549627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.549638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.549700 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:11.549876 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.550603 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.552008 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.552625 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.552697 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.552706 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.552936 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.552985 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:53:11.553071 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.549436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:44825 (local address 127.3.0.195:54613)
0928 21:53:11.549490 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:11.549503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:11.549562 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:11.549717 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:11.549718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:11.549720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:11.549800 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.549803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.550684 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.550685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.551899 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.551903 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.552709 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.552710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.552809 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.552816 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:53:11.552875 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:53:11.552975 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":3}
I20260928 21:53:11.553081 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.549466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:39141 (local address 127.3.0.195:53179)
0928 21:53:11.549652 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:11.549658 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:11.549702 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:11.549823 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:11.549824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:11.549825 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:11.549915 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.549916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.550759 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.550760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.551917 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.551918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.552721 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.552722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.552809 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.552816 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:53:11.552875 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:53:11.552975 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":69,"thread_start_us":111,"threads_started":1}
I20260928 21:53:11.553520 20251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aae79459b684cc08662596cb8d8bbc2" candidate_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4770db0c5614bbaae7128e40b32300b" is_pre_election: true
I20260928 21:53:11.553680 20251 raft_consensus.cc:2454] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded914ade3a244c88ba5cd5ef1c2c13e in term 0.
I20260928 21:53:11.553989 20452 leader_election.cc:302] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4770db0c5614bbaae7128e40b32300b, ded914ade3a244c88ba5cd5ef1c2c13e; no voters:
I20260928 21:53:11.554024 20385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aae79459b684cc08662596cb8d8bbc2" candidate_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33e2437df314c358beeea8e516a82d6" is_pre_election: true
I20260928 21:53:11.554171 20385 raft_consensus.cc:2454] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ded914ade3a244c88ba5cd5ef1c2c13e in term 0.
I20260928 21:53:11.554157 20585 raft_consensus.cc:2781] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:53:11.554229 20585 raft_consensus.cc:482] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:53:11.554247 20585 raft_consensus.cc:3037] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:11.554980 20585 raft_consensus.cc:504] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 } } peers { permanent_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 38139 } } peers { permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 } }
I20260928 21:53:11.555109 20585 leader_election.cc:288] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [CANDIDATE]: Term 1 election: Requested vote from peers a4770db0c5614bbaae7128e40b32300b (127.3.0.193:44825), f33e2437df314c358beeea8e516a82d6 (127.3.0.194:39141)
I20260928 21:53:11.555295 20385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aae79459b684cc08662596cb8d8bbc2" candidate_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33e2437df314c358beeea8e516a82d6"
I20260928 21:53:11.555295 20251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aae79459b684cc08662596cb8d8bbc2" candidate_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4770db0c5614bbaae7128e40b32300b"
I20260928 21:53:11.555387 20385 raft_consensus.cc:3037] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:11.555387 20251 raft_consensus.cc:3037] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:11.556310 20385 raft_consensus.cc:2454] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded914ade3a244c88ba5cd5ef1c2c13e in term 1.
I20260928 21:53:11.556310 20251 raft_consensus.cc:2454] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ded914ade3a244c88ba5cd5ef1c2c13e in term 1.
I20260928 21:53:11.556586 20454 leader_election.cc:302] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ded914ade3a244c88ba5cd5ef1c2c13e, f33e2437df314c358beeea8e516a82d6; no voters:
I20260928 21:53:11.556674 20585 raft_consensus.cc:2781] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:11.556838 20585 raft_consensus.cc:686] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 1 LEADER]: Becoming Leader. State: Replica: ded914ade3a244c88ba5cd5ef1c2c13e, State: Running, Role: LEADER
I20260928 21:53:11.556972 20585 consensus_queue.cc:237] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 } } peers { permanent_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 38139 } } peers { permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 } }
I20260928 21:53:11.557801 20105 catalog_manager.cc:5495] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e reported cstate change: term changed from 0 to 1, leader changed from <none> to ded914ade3a244c88ba5cd5ef1c2c13e (127.3.0.195). New cstate: current_term: 1 leader_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded914ade3a244c88ba5cd5ef1c2c13e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 38139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 } health_report { overall_health: UNKNOWN } } }
I20260928 21:53:11.567183 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.564729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36032 (local address 127.3.0.195:38139)
0928 21:53:11.564782 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.564784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.564796 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:53:11.564832 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.564833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.564852 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.564915 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:11.565035 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.565573 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.566347 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.566554 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.566609 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.566844 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:11.566982 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:11.567056 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.567079 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.567102 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:53:11.568998 20385 raft_consensus.cc:1261] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6 [term 1 FOLLOWER]: Refusing update from remote peer ded914ade3a244c88ba5cd5ef1c2c13e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:11.568984 20251 raft_consensus.cc:1261] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b [term 1 FOLLOWER]: Refusing update from remote peer ded914ade3a244c88ba5cd5ef1c2c13e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:11.569519 20585 consensus_queue.cc:1035] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4770db0c5614bbaae7128e40b32300b" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:11.569671 20589 consensus_queue.cc:1035] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f33e2437df314c358beeea8e516a82d6" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 39141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:11.572268 20594 mvcc.cc:205] Tried to move back new op lower bound from 7334430275863760896 to 7334430275818139648. Current Snapshot: MvccSnapshot[applied={T|T < 7334430275863760896}]
I20260928 21:53:11.572558 20593 mvcc.cc:205] Tried to move back new op lower bound from 7334430275863760896 to 7334430275818139648. Current Snapshot: MvccSnapshot[applied={T|T < 7334430275863760896}]
I20260928 21:53:11.574491 20106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:11.574573 20106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:11.575441 20106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a5f9f43aa0542f3af310b7e512c5104: Sending DeleteTablet for 3 replicas of tablet 2aae79459b684cc08662596cb8d8bbc2
I20260928 21:53:11.575753 20231 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aae79459b684cc08662596cb8d8bbc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:11 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:44428
I20260928 21:53:11.575996 20602 tablet_replica.cc:330] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b: stopping tablet replica
I20260928 21:53:11.576000 20499 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aae79459b684cc08662596cb8d8bbc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:11 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:36026
I20260928 21:53:11.576103 20602 raft_consensus.cc:2229] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:11.576211 20602 raft_consensus.cc:2258] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:11.576349 20365 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aae79459b684cc08662596cb8d8bbc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:11 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:36462
I20260928 21:53:11.576653 20602 ts_tablet_manager.cc:1905] T 2aae79459b684cc08662596cb8d8bbc2 P a4770db0c5614bbaae7128e40b32300b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:11.576725 20603 tablet_replica.cc:330] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: stopping tablet replica
I20260928 21:53:11.576828 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 20168
I20260928 21:53:11.577412 20604 tablet_replica.cc:330] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: stopping tablet replica
I20260928 21:53:11.577522 20604 raft_consensus.cc:2229] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:11.577622 20604 raft_consensus.cc:2258] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:11.577754 20603 raft_consensus.cc:2229] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:53:11.577853 20603 raft_consensus.cc:2258] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:11.578441 20604 ts_tablet_manager.cc:1905] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:11.580183 20604 ts_tablet_manager.cc:1918] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:11.580276 20603 ts_tablet_manager.cc:1905] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:11.580302 20604 log.cc:1196] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/2aae79459b684cc08662596cb8d8bbc2
I20260928 21:53:11.580668 20604 ts_tablet_manager.cc:1939] T 2aae79459b684cc08662596cb8d8bbc2 P f33e2437df314c358beeea8e516a82d6: Deleting consensus metadata
I20260928 21:53:11.581182 20092 catalog_manager.cc:4841] TS f33e2437df314c358beeea8e516a82d6 (127.3.0.194:39141): tablet 2aae79459b684cc08662596cb8d8bbc2 (table test-table [id=f6a12fe0197c471cabdaec72bfca7ab0]) successfully deleted
I20260928 21:53:11.581990 20603 ts_tablet_manager.cc:1918] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:11.582065 20603 log.cc:1196] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/2aae79459b684cc08662596cb8d8bbc2
I20260928 21:53:11.582321 20603 ts_tablet_manager.cc:1939] T 2aae79459b684cc08662596cb8d8bbc2 P ded914ade3a244c88ba5cd5ef1c2c13e: Deleting consensus metadata
W20260928 21:53:11.582639 20090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:44825 (error 108)
I20260928 21:53:11.582904 20090 catalog_manager.cc:4841] TS ded914ade3a244c88ba5cd5ef1c2c13e (127.3.0.195:38139): tablet 2aae79459b684cc08662596cb8d8bbc2 (table test-table [id=f6a12fe0197c471cabdaec72bfca7ab0]) successfully deleted
W20260928 21:53:11.583211 20576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:11.583066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:44825 (local address 127.0.0.1:44438)
0928 21:53:11.583114 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:11.583126 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:11.583167 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:44825: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 21:53:11.583312 20090 catalog_manager.cc:4584] TS a4770db0c5614bbaae7128e40b32300b (127.3.0.193:44825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2aae79459b684cc08662596cb8d8bbc2: Network error: Client connection negotiation failed: client connection to 127.3.0.193:44825: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 21:53:11.583566 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 20302
I20260928 21:53:11.587991 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 20436
I20260928 21:53:11.592293 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 20074
2026-09-28T21:53:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1780 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T21:53:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:11Z Disabled control of system clock
I20260928 21:53:11.618680 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:39425
--webserver_interface=127.3.0.254
--webserver_port=0
--builtin_ntp_servers=127.3.0.212:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:39425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:11.697875 20609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:11.698102 20609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:11.698122 20609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:11.699255 20609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:11.699290 20609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:11.699301 20609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:11.699311 20609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:11.699321 20609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:11.700646 20609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:39425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:39425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:11.700835 20609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:11.701076 20609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:11.705678 20614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:11.705719 20609 server_base.cc:1034] running on GCE node
W20260928 21:53:11.705720 20615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:11.705682 20617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:11.706126 20609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:11.706425 20609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:11.707597 20609 hybrid_clock.cc:648] HybridClock initialized: now 1790632391707576 us; error 30 us; skew 500 ppm
I20260928 21:53:11.708969 20609 webserver.cc:460] Webserver started at http://127.3.0.254:40615/ using document root <none> and password file <none>
I20260928 21:53:11.709199 20609 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:11.709250 20609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:11.709380 20609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:11.710438 20609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "4e78466b71054d3fa6281b929983cc31"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "4e6e68c1ff723aec9e37a5efc9f7212e"
server_key_iv: "dfb6cf1968500ef90f554c0bc83ad771"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.710754 20609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "4e78466b71054d3fa6281b929983cc31"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "4e6e68c1ff723aec9e37a5efc9f7212e"
server_key_iv: "dfb6cf1968500ef90f554c0bc83ad771"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.711939 20609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:11.712610 20623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.712745 20609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:11.712826 20609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "4e78466b71054d3fa6281b929983cc31"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "4e6e68c1ff723aec9e37a5efc9f7212e"
server_key_iv: "dfb6cf1968500ef90f554c0bc83ad771"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.712891 20609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:11.732820 20609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:11.733129 20609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:11.733232 20609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:11.736811 20609 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:39425
I20260928 21:53:11.736848 20675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:39425 every 8 connection(s)
I20260928 21:53:11.737167 20609 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:11.738101 20676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:11.740443 20676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31: Bootstrap starting.
I20260928 21:53:11.741017 20676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:11.741281 20676 log.cc:824] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:11.741904 20676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31: No bootstrap required, opened a new log
I20260928 21:53:11.742775 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 20609
I20260928 21:53:11.742852 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:11.743021 3075 external_mini_cluster.cc:1428] Setting key 644442ebd55810c6b41d8fc5e3dd0b04
I20260928 21:53:11.743716 20676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e78466b71054d3fa6281b929983cc31" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39425 } }
I20260928 21:53:11.743891 20676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:11.743939 20676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e78466b71054d3fa6281b929983cc31, State: Initialized, Role: FOLLOWER
I20260928 21:53:11.744061 20676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e78466b71054d3fa6281b929983cc31" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39425 } }
I20260928 21:53:11.744139 20676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:11.744168 20676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:11.744211 20676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:11.744766 20676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e78466b71054d3fa6281b929983cc31" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39425 } }
I20260928 21:53:11.744855 20676 leader_election.cc:302] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e78466b71054d3fa6281b929983cc31; no voters:
I20260928 21:53:11.745009 20676 leader_election.cc:288] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:11.745056 20681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:11.745165 20681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [term 1 LEADER]: Becoming Leader. State: Replica: 4e78466b71054d3fa6281b929983cc31, State: Running, Role: LEADER
I20260928 21:53:11.745262 20676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:11.745244 20681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e78466b71054d3fa6281b929983cc31" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39425 } }
I20260928 21:53:11.745610 20681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e78466b71054d3fa6281b929983cc31. Latest consensus state: current_term: 1 leader_uuid: "4e78466b71054d3fa6281b929983cc31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e78466b71054d3fa6281b929983cc31" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39425 } } }
I20260928 21:53:11.745694 20681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:11.745618 20683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e78466b71054d3fa6281b929983cc31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e78466b71054d3fa6281b929983cc31" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 39425 } } }
I20260928 21:53:11.745836 20683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:11.745913 20687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:11.746393 20687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:11.748588 20687 catalog_manager.cc:1347] Generated new cluster ID: d208537d42cd4109ad30b45b09cc97dd
I20260928 21:53:11.748651 20687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:11.748852 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.743494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.3.0.254:39425)
0928 21:53:11.743717 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.743727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.743763 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:53:11.743883 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.743890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.743962 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.744100 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:11.744313 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.745882 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.746926 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.747475 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.747632 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.748218 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:11.748222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:11.748222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:11.748387 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:11.748419 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:11.748430 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:11.748514 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.748584 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.748662 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260928 21:53:11.771425 20687 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:11.772140 20687 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:11.780539 20687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31: Generated new TSK 0
I20260928 21:53:11.780730 20687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:11.785383 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39425
--builtin_ntp_servers=127.3.0.212:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:11.864490 20700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:11.864694 20700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:11.864715 20700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:11.865851 20700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:11.865888 20700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:11.865911 20700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:53:11.867220 20700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:11.867448 20700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:11.867727 20700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:11.873325 20705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:11.873335 20706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:11.873515 20708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:11.873652 20700 server_base.cc:1034] running on GCE node
I20260928 21:53:11.873831 20700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:11.874051 20700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:11.875190 20700 hybrid_clock.cc:648] HybridClock initialized: now 1790632391875174 us; error 29 us; skew 500 ppm
I20260928 21:53:11.876335 20700 webserver.cc:460] Webserver started at http://127.3.0.193:40109/ using document root <none> and password file <none>
I20260928 21:53:11.876600 20700 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:11.876650 20700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:11.876771 20700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:11.877799 20700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "ead5cebffee743d68bc81395c2bd3d02"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "05b58d9a9eb9761e2e74d1bda03d8020"
server_key_iv: "ae490e91bd0f9395f7b8139229295951"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.878124 20700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "ead5cebffee743d68bc81395c2bd3d02"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "05b58d9a9eb9761e2e74d1bda03d8020"
server_key_iv: "ae490e91bd0f9395f7b8139229295951"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.879305 20700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:11.879971 20714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.880127 20700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:11.880204 20700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "ead5cebffee743d68bc81395c2bd3d02"
format_stamp: "Formatted at 2026-09-28 21:53:11 on dist-test-slave-jrck"
server_key: "05b58d9a9eb9761e2e74d1bda03d8020"
server_key_iv: "ae490e91bd0f9395f7b8139229295951"
server_key_version: "encryptionkey@0"
I20260928 21:53:11.880268 20700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:11.893535 20700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:11.893823 20700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:11.893957 20700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:11.894166 20700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:11.894484 20700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:11.894526 20700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.894552 20700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:11.894568 20700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:11.900056 20700 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:42445
I20260928 21:53:11.900110 20827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:42445 every 8 connection(s)
I20260928 21:53:11.900403 20700 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:53:11.905215 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.901073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:51595 (local address 127.3.0.254:39425)
0928 21:53:11.901131 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:11.901133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:11.901359 (+ 226us) server_negotiation.cc:407] Connection header received
0928 21:53:11.901819 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:11.901820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:11.901832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:11.901922 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:11.902581 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.903090 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.903977 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.904177 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.904226 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.904984 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:11.904986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:11.904986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:11.904990 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:11.905006 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:11.905009 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:11.905061 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:11.905129 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 21:53:11.905157 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:53:11.905391 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:11.901048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:39425 (local address 127.3.0.193:51595)
0928 21:53:11.901237 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:11.901277 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:11.901740 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:11.901985 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:11.901988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:11.902019 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:11.902513 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.902518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.903155 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.903155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.903910 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:11.903912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:11.904237 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:11.904238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:11.904340 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:11.904830 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:53:11.904830 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:11.904859 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:53:11.904881 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:11.905067 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:11.905070 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:11.905073 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:11.905119 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:53:11.905196 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260928 21:53:11.905978 20828 heartbeater.cc:347] Connected to a master server at 127.3.0.254:39425
I20260928 21:53:11.906078 20828 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:11.906276 20828 heartbeater.cc:511] Master 127.3.0.254:39425 requested a full tablet report, sending...
I20260928 21:53:11.906802 20640 ts_manager.cc:194] Registered new tserver with Master: ead5cebffee743d68bc81395c2bd3d02 (127.3.0.193:42445)
I20260928 21:53:11.907398 20640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.193:51595
I20260928 21:53:11.909937 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 20700
I20260928 21:53:11.910028 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:53:11.910214 3075 external_mini_cluster.cc:1428] Setting key 2f9fa7b0b4935c34045efb978a17aa0a
I20260928 21:53:11.911024 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39425
--builtin_ntp_servers=127.3.0.212:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:11.989926 20831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:11.990123 20831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:11.990144 20831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:11.991226 20831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:11.991258 20831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:11.991281 20831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:53:11.992586 20831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:11.992771 20831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:11.993012 20831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:11.997701 20836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:11.997701 20837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:11.997697 20839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:11.997998 20831 server_base.cc:1034] running on GCE node
I20260928 21:53:11.998234 20831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:11.998481 20831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:11.999641 20831 hybrid_clock.cc:648] HybridClock initialized: now 1790632391999627 us; error 30 us; skew 500 ppm
I20260928 21:53:12.000975 20831 webserver.cc:460] Webserver started at http://127.3.0.194:39597/ using document root <none> and password file <none>
I20260928 21:53:12.001216 20831 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:12.001266 20831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:12.001391 20831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:12.002439 20831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "9e5a3332088146f0abd880571a549dd7"
format_stamp: "Formatted at 2026-09-28 21:53:12 on dist-test-slave-jrck"
server_key: "361d55d23e51ab7ee50020447089c3c3"
server_key_iv: "8017842ecd12ce2037b6de0cdb95c1be"
server_key_version: "encryptionkey@0"
I20260928 21:53:12.002770 20831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "9e5a3332088146f0abd880571a549dd7"
format_stamp: "Formatted at 2026-09-28 21:53:12 on dist-test-slave-jrck"
server_key: "361d55d23e51ab7ee50020447089c3c3"
server_key_iv: "8017842ecd12ce2037b6de0cdb95c1be"
server_key_version: "encryptionkey@0"
I20260928 21:53:12.003968 20831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:12.004756 20845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:12.004926 20831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:12.004999 20831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "9e5a3332088146f0abd880571a549dd7"
format_stamp: "Formatted at 2026-09-28 21:53:12 on dist-test-slave-jrck"
server_key: "361d55d23e51ab7ee50020447089c3c3"
server_key_iv: "8017842ecd12ce2037b6de0cdb95c1be"
server_key_version: "encryptionkey@0"
I20260928 21:53:12.005066 20831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:12.022084 20831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:12.022392 20831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:12.022528 20831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:12.022748 20831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:12.023078 20831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:12.023111 20831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:12.023140 20831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:12.023157 20831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:12.028626 20831 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:37603
I20260928 21:53:12.028678 20958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:37603 every 8 connection(s)
I20260928 21:53:12.028997 20831 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:53:12.034276 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.029763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:56401 (local address 127.3.0.254:39425)
0928 21:53:12.029821 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:12.029823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:12.030051 (+ 228us) server_negotiation.cc:407] Connection header received
0928 21:53:12.030583 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:12.030584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:12.030595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:12.030649 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:12.031315 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.031822 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.032958 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.033166 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.033229 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.034013 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:12.034014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:12.034014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:12.034019 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:12.034032 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:12.034036 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:12.034076 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:12.034177 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 21:53:12.034209 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:53:12.034437 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.029757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:39425 (local address 127.3.0.194:56401)
0928 21:53:12.029931 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:12.029966 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:12.030473 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:12.030707 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:12.030710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:12.030720 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:12.031216 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.031221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.031900 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.031901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.032848 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.032851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.033242 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.033242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.033357 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.033875 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:53:12.033876 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:12.033906 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:53:12.033934 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:12.034078 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:12.034081 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:12.034083 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:53:12.034140 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:53:12.034232 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260928 21:53:12.034924 20959 heartbeater.cc:347] Connected to a master server at 127.3.0.254:39425
I20260928 21:53:12.035036 20959 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:12.035223 20959 heartbeater.cc:511] Master 127.3.0.254:39425 requested a full tablet report, sending...
I20260928 21:53:12.035571 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 20831
I20260928 21:53:12.035668 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:53:12.035673 20640 ts_manager.cc:194] Registered new tserver with Master: 9e5a3332088146f0abd880571a549dd7 (127.3.0.194:37603)
I20260928 21:53:12.035915 3075 external_mini_cluster.cc:1428] Setting key 1c377ff8147b8154cf2a0a6e5aa3e9e9
I20260928 21:53:12.036070 20640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.194:56401
I20260928 21:53:12.036816 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39425
--builtin_ntp_servers=127.3.0.212:42663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:12.118863 20962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:12.119062 20962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:12.119082 20962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:12.120136 20962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:12.120167 20962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:12.120190 20962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:53:12.121517 20962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:12.121693 20962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:12.121924 20962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:12.126457 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:12.126484 20970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:12.126467 20968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:12.126612 20962 server_base.cc:1034] running on GCE node
I20260928 21:53:12.126813 20962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:12.127027 20962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:12.128157 20962 hybrid_clock.cc:648] HybridClock initialized: now 1790632392128150 us; error 33 us; skew 500 ppm
I20260928 21:53:12.129324 20962 webserver.cc:460] Webserver started at http://127.3.0.195:34233/ using document root <none> and password file <none>
I20260928 21:53:12.129532 20962 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:12.129577 20962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:12.129686 20962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:12.130677 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "d274e7de5ead4b23bad53035c811b4b7"
format_stamp: "Formatted at 2026-09-28 21:53:12 on dist-test-slave-jrck"
server_key: "4722f2508bda859085c84c8cb03588c3"
server_key_iv: "19d0e84a00de6525e00f5f74d91e8160"
server_key_version: "encryptionkey@0"
I20260928 21:53:12.130982 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "d274e7de5ead4b23bad53035c811b4b7"
format_stamp: "Formatted at 2026-09-28 21:53:12 on dist-test-slave-jrck"
server_key: "4722f2508bda859085c84c8cb03588c3"
server_key_iv: "19d0e84a00de6525e00f5f74d91e8160"
server_key_version: "encryptionkey@0"
I20260928 21:53:12.132155 20962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:12.132869 20976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:12.133003 20962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:12.133080 20962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "d274e7de5ead4b23bad53035c811b4b7"
format_stamp: "Formatted at 2026-09-28 21:53:12 on dist-test-slave-jrck"
server_key: "4722f2508bda859085c84c8cb03588c3"
server_key_iv: "19d0e84a00de6525e00f5f74d91e8160"
server_key_version: "encryptionkey@0"
I20260928 21:53:12.133145 20962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:12.144134 20962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:12.144438 20962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:12.144609 20962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:12.144829 20962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:12.145131 20962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:12.145164 20962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:12.145192 20962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:12.145212 20962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:12.150548 20962 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:43415
I20260928 21:53:12.150609 21089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:43415 every 8 connection(s)
I20260928 21:53:12.150892 20962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:53:12.151413 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 20962
I20260928 21:53:12.151507 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:53:12.151715 3075 external_mini_cluster.cc:1428] Setting key 6d08d87aa1f0afbaafe266a69a1fa2e9
I20260928 21:53:12.155936 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.151562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:55469 (local address 127.3.0.254:39425)
0928 21:53:12.151615 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:12.151618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:12.151902 (+ 284us) server_negotiation.cc:407] Connection header received
0928 21:53:12.152429 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:12.152430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:12.152438 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:12.152500 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:12.153215 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.153726 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.154639 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.154832 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.154936 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.155687 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:12.155688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:12.155688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:12.155693 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:12.155707 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:12.155711 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:12.155754 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:12.155820 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 21:53:12.155846 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:53:12.156126 21092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.151563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:39425 (local address 127.3.0.195:55469)
0928 21:53:12.151792 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:12.151831 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:12.152341 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:12.152588 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:12.152592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:12.152602 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:12.153125 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.153130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.153793 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.153794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.154560 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.154562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.154941 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.154942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.155067 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.155552 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:53:12.155553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:12.155581 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:53:12.155604 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:12.155763 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:12.155766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:12.155769 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:12.155816 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:53:12.155895 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260928 21:53:12.156561 21090 heartbeater.cc:347] Connected to a master server at 127.3.0.254:39425
I20260928 21:53:12.156648 21090 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:12.156814 21090 heartbeater.cc:511] Master 127.3.0.254:39425 requested a full tablet report, sending...
I20260928 21:53:12.157135 20640 ts_manager.cc:194] Registered new tserver with Master: d274e7de5ead4b23bad53035c811b4b7 (127.3.0.195:43415)
I20260928 21:53:12.157465 20640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.195:55469
I20260928 21:53:12.163605 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:53:12.167760 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.165058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.3.0.254:39425)
0928 21:53:12.165118 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:12.165120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:12.165177 (+ 57us) server_negotiation.cc:407] Connection header received
0928 21:53:12.165256 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:12.165257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:12.165270 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:12.165314 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:12.165471 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.165944 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.166671 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.166852 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.166905 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.167524 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:12.167526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:12.167527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:12.167531 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:12.167544 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:12.167548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:12.167589 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:12.167649 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 21:53:12.167677 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:53:12.169268 20640 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:53:12.170783 20640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:53:12.176980 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.173035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.3.0.194:37603)
0928 21:53:12.173216 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:12.173221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:12.173264 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:53:12.173340 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:12.173342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:12.173353 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:12.173587 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:12.173647 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.174764 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.176052 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.176588 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.176652 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.176745 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:12.176851 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 21:53:12.176888 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260928 21:53:12.177039 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.173067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.3.0.195:43415)
0928 21:53:12.173535 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:12.173539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:12.173548 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:53:12.173581 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:12.173583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:12.173597 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:12.173662 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:12.173833 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.174516 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.175941 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.176639 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.176691 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.176759 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:12.176922 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 21:53:12.176957 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":69,"threads_started":1}
I20260928 21:53:12.177204 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.172830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:42445 (local address 127.0.0.1:47460)
0928 21:53:12.172993 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:12.173035 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:12.173099 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:12.173425 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:12.173428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:12.173430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:12.173518 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.173520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.174873 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.174873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.175758 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.175760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.176978 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.176979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.177045 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.177049 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:53:12.177104 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:53:12.177141 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":19,"thread_start_us":106,"threads_started":1}
I20260928 21:53:12.177358 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.172830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:37603 (local address 127.0.0.1:55328)
0928 21:53:12.173203 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:12.173214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:12.173277 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:12.173485 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:12.173486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:12.173487 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:12.173521 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.173522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.174857 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.174858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.175981 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.175983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.176661 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.176662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.176749 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.176764 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:53:12.176826 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:53:12.176889 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":33,"thread_start_us":57,"threads_started":1}
I20260928 21:53:12.177436 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.173028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.3.0.193:42445)
0928 21:53:12.173235 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:12.173240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:12.173250 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:53:12.173291 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:12.173293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:12.173310 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:12.173416 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:12.173649 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.174763 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.175829 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.176585 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.176658 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.176748 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:12.177304 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 21:53:12.177350 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20260928 21:53:12.177999 20762 tablet_service.cc:1467] Processing CreateTablet for tablet 6af0796d5e254956bc3ae4087a524698 (DEFAULT_TABLE table=test-table [id=61dfa76881484496ab85c6659bb77441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:12.178020 20893 tablet_service.cc:1467] Processing CreateTablet for tablet 6af0796d5e254956bc3ae4087a524698 (DEFAULT_TABLE table=test-table [id=61dfa76881484496ab85c6659bb77441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:12.178282 20762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af0796d5e254956bc3ae4087a524698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:12.178299 20893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af0796d5e254956bc3ae4087a524698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:12.180671 21105 tablet_bootstrap.cc:492] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: Bootstrap starting.
I20260928 21:53:12.180828 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.172842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:43415 (local address 127.0.0.1:45994)
0928 21:53:12.173359 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:12.173367 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:12.173400 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:12.173723 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:12.173724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:12.173726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:12.173769 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.173770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.174631 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.174632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.175863 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.175865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.176791 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.176792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.176847 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.176850 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:12.176897 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:53:12.176939 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":98,"threads_started":1}
I20260928 21:53:12.181058 21104 tablet_bootstrap.cc:492] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02: Bootstrap starting.
I20260928 21:53:12.181316 21105 tablet_bootstrap.cc:654] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:12.181625 21104 tablet_bootstrap.cc:654] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:12.181631 21105 log.cc:824] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:12.181528 21024 tablet_service.cc:1467] Processing CreateTablet for tablet 6af0796d5e254956bc3ae4087a524698 (DEFAULT_TABLE table=test-table [id=61dfa76881484496ab85c6659bb77441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:12.181845 21024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af0796d5e254956bc3ae4087a524698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:12.181910 21104 log.cc:824] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:12.182353 21105 tablet_bootstrap.cc:492] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: No bootstrap required, opened a new log
I20260928 21:53:12.182426 21105 ts_tablet_manager.cc:1397] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:53:12.182570 21104 tablet_bootstrap.cc:492] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02: No bootstrap required, opened a new log
I20260928 21:53:12.182628 21104 ts_tablet_manager.cc:1397] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:53:12.184646 21104 raft_consensus.cc:348] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5a3332088146f0abd880571a549dd7" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37603 } } peers { permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 } } peers { permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 } }
I20260928 21:53:12.184774 21104 raft_consensus.cc:374] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:12.184804 21104 raft_consensus.cc:729] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead5cebffee743d68bc81395c2bd3d02, State: Initialized, Role: FOLLOWER
I20260928 21:53:12.184933 21104 consensus_queue.cc:260] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5a3332088146f0abd880571a549dd7" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37603 } } peers { permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 } } peers { permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 } }
I20260928 21:53:12.184859 21105 raft_consensus.cc:348] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5a3332088146f0abd880571a549dd7" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37603 } } peers { permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 } } peers { permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 } }
I20260928 21:53:12.185004 21105 raft_consensus.cc:374] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:12.185037 21105 raft_consensus.cc:729] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5a3332088146f0abd880571a549dd7, State: Initialized, Role: FOLLOWER
I20260928 21:53:12.185142 21105 consensus_queue.cc:260] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5a3332088146f0abd880571a549dd7" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37603 } } peers { permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 } } peers { permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 } }
I20260928 21:53:12.185187 21108 tablet_bootstrap.cc:492] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: Bootstrap starting.
I20260928 21:53:12.185218 21104 ts_tablet_manager.cc:1428] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:53:12.185328 20828 heartbeater.cc:503] Master 127.3.0.254:39425 was elected leader, sending a full tablet report...
I20260928 21:53:12.185382 21105 ts_tablet_manager.cc:1428] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:53:12.185509 20959 heartbeater.cc:503] Master 127.3.0.254:39425 was elected leader, sending a full tablet report...
I20260928 21:53:12.185793 21108 tablet_bootstrap.cc:654] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:12.186096 21108 log.cc:824] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:12.186797 21108 tablet_bootstrap.cc:492] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: No bootstrap required, opened a new log
I20260928 21:53:12.186853 21108 ts_tablet_manager.cc:1397] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:53:12.188736 21108 raft_consensus.cc:348] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5a3332088146f0abd880571a549dd7" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37603 } } peers { permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 } } peers { permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 } }
I20260928 21:53:12.188858 21108 raft_consensus.cc:374] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:12.188886 21108 raft_consensus.cc:729] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d274e7de5ead4b23bad53035c811b4b7, State: Initialized, Role: FOLLOWER
I20260928 21:53:12.188992 21108 consensus_queue.cc:260] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5a3332088146f0abd880571a549dd7" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37603 } } peers { permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 } } peers { permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 } }
I20260928 21:53:12.189201 21108 ts_tablet_manager.cc:1428] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:53:12.189247 21090 heartbeater.cc:503] Master 127.3.0.254:39425 was elected leader, sending a full tablet report...
I20260928 21:53:12.223634 21110 raft_consensus.cc:482] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:53:12.223754 21110 raft_consensus.cc:504] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5a3332088146f0abd880571a549dd7" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37603 } } peers { permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 } } peers { permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 } }
I20260928 21:53:12.224066 21110 leader_election.cc:288] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ead5cebffee743d68bc81395c2bd3d02 (127.3.0.193:42445), d274e7de5ead4b23bad53035c811b4b7 (127.3.0.195:43415)
I20260928 21:53:12.226864 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.224290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:51065 (local address 127.3.0.195:43415)
0928 21:53:12.224339 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:12.224341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:12.224371 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:53:12.224401 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:12.224402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:12.224409 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:12.224484 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:12.224633 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.225174 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.226141 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.226569 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.226617 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.226625 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:12.226769 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 21:53:12.226795 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:53:12.226878 20961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.224223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:43415 (local address 127.3.0.194:51065)
0928 21:53:12.224284 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:12.224296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:12.224336 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:12.224506 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:12.224507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:12.224509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:12.224573 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.224574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.225247 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.225248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.226060 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.226063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.226638 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.226639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.226705 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.226709 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:53:12.226766 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:53:12.226806 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 21:53:12.227144 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.224288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:40669 (local address 127.3.0.193:42445)
0928 21:53:12.224341 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:12.224343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:12.224623 (+ 280us) server_negotiation.cc:407] Connection header received
0928 21:53:12.224655 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:12.224656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:12.224668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:12.224707 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:12.224878 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.225395 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.226344 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.226765 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.226819 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.226827 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:12.227024 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:53:12.227078 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:53:12.227232 21044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af0796d5e254956bc3ae4087a524698" candidate_uuid: "9e5a3332088146f0abd880571a549dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d274e7de5ead4b23bad53035c811b4b7" is_pre_election: true
I20260928 21:53:12.227358 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.224228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:42445 (local address 127.3.0.194:40669)
0928 21:53:12.224551 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:12.224561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:12.224608 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:12.224720 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:12.224721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:12.224722 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:12.224786 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.224787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.225467 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.225468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.226260 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:12.226262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.226844 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.226844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:12.226920 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.226924 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:53:12.227251 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 21:53:12.227292 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":146,"thread_start_us":114,"threads_started":1}
I20260928 21:53:12.227398 21044 raft_consensus.cc:2454] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e5a3332088146f0abd880571a549dd7 in term 0.
I20260928 21:53:12.227617 20848 leader_election.cc:302] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e5a3332088146f0abd880571a549dd7, d274e7de5ead4b23bad53035c811b4b7; no voters:
I20260928 21:53:12.227671 20782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af0796d5e254956bc3ae4087a524698" candidate_uuid: "9e5a3332088146f0abd880571a549dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead5cebffee743d68bc81395c2bd3d02" is_pre_election: true
I20260928 21:53:12.227741 21110 raft_consensus.cc:2781] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:53:12.227803 21110 raft_consensus.cc:482] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:53:12.227813 20782 raft_consensus.cc:2454] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e5a3332088146f0abd880571a549dd7 in term 0.
I20260928 21:53:12.227833 21110 raft_consensus.cc:3037] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:12.228528 21110 raft_consensus.cc:504] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5a3332088146f0abd880571a549dd7" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37603 } } peers { permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 } } peers { permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 } }
I20260928 21:53:12.228652 21110 leader_election.cc:288] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [CANDIDATE]: Term 1 election: Requested vote from peers ead5cebffee743d68bc81395c2bd3d02 (127.3.0.193:42445), d274e7de5ead4b23bad53035c811b4b7 (127.3.0.195:43415)
I20260928 21:53:12.228840 21044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af0796d5e254956bc3ae4087a524698" candidate_uuid: "9e5a3332088146f0abd880571a549dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d274e7de5ead4b23bad53035c811b4b7"
I20260928 21:53:12.228801 20782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af0796d5e254956bc3ae4087a524698" candidate_uuid: "9e5a3332088146f0abd880571a549dd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead5cebffee743d68bc81395c2bd3d02"
I20260928 21:53:12.228907 21044 raft_consensus.cc:3037] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:12.228910 20782 raft_consensus.cc:3037] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:12.229559 21044 raft_consensus.cc:2454] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e5a3332088146f0abd880571a549dd7 in term 1.
I20260928 21:53:12.229703 20782 raft_consensus.cc:2454] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e5a3332088146f0abd880571a549dd7 in term 1.
I20260928 21:53:12.229756 20848 leader_election.cc:302] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e5a3332088146f0abd880571a549dd7, d274e7de5ead4b23bad53035c811b4b7; no voters:
I20260928 21:53:12.229830 21110 raft_consensus.cc:2781] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:12.229975 21110 raft_consensus.cc:686] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 1 LEADER]: Becoming Leader. State: Replica: 9e5a3332088146f0abd880571a549dd7, State: Running, Role: LEADER
I20260928 21:53:12.230077 21110 consensus_queue.cc:237] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5a3332088146f0abd880571a549dd7" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37603 } } peers { permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 } } peers { permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 } }
I20260928 21:53:12.230842 20639 catalog_manager.cc:5495] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e5a3332088146f0abd880571a549dd7 (127.3.0.194). New cstate: current_term: 1 leader_uuid: "9e5a3332088146f0abd880571a549dd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e5a3332088146f0abd880571a549dd7" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 37603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 } health_report { overall_health: UNKNOWN } } }
I20260928 21:53:12.250401 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.247767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.3.0.194:37603)
0928 21:53:12.247809 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:12.247811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:12.247822 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:53:12.247891 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:12.247892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:12.247913 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:12.247988 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:12.248119 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.248842 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.249675 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:12.249882 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:12.249939 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:12.250077 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:12.250190 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:12.250262 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:12.250280 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 21:53:12.250300 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:53:12.252641 21044 raft_consensus.cc:1261] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7 [term 1 FOLLOWER]: Refusing update from remote peer 9e5a3332088146f0abd880571a549dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:12.253090 21110 consensus_queue.cc:1035] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d274e7de5ead4b23bad53035c811b4b7" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:12.254087 20782 raft_consensus.cc:1261] T 6af0796d5e254956bc3ae4087a524698 P ead5cebffee743d68bc81395c2bd3d02 [term 1 FOLLOWER]: Refusing update from remote peer 9e5a3332088146f0abd880571a549dd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:12.254534 21114 consensus_queue.cc:1035] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ead5cebffee743d68bc81395c2bd3d02" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:12.255968 21118 mvcc.cc:205] Tried to move back new op lower bound from 7334430278663839744 to 7334430278575013888. Current Snapshot: MvccSnapshot[applied={T|T < 7334430278663839744 or (T in {7334430278663839744})}]
I20260928 21:53:12.256189 21117 mvcc.cc:205] Tried to move back new op lower bound from 7334430278663839744 to 7334430278575013888. Current Snapshot: MvccSnapshot[applied={T|T < 7334430278663839744}]
I20260928 21:53:12.257087 21122 mvcc.cc:205] Tried to move back new op lower bound from 7334430278663839744 to 7334430278575013888. Current Snapshot: MvccSnapshot[applied={T|T < 7334430278663839744}]
I20260928 21:53:12.258401 20639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:12.258484 20639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:12.259243 20639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e78466b71054d3fa6281b929983cc31: Sending DeleteTablet for 3 replicas of tablet 6af0796d5e254956bc3ae4087a524698
I20260928 21:53:12.259640 20893 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af0796d5e254956bc3ae4087a524698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:12 UTC) from {username='slave'} at 127.0.0.1:55328
I20260928 21:53:12.259667 20762 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af0796d5e254956bc3ae4087a524698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:12 UTC) from {username='slave'} at 127.0.0.1:47460
I20260928 21:53:12.259831 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 20700
I20260928 21:53:12.259836 21024 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af0796d5e254956bc3ae4087a524698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:12 UTC) from {username='slave'} at 127.0.0.1:45994
I20260928 21:53:12.260721 21126 tablet_replica.cc:330] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: stopping tablet replica
I20260928 21:53:12.260864 21126 raft_consensus.cc:2229] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:53:12.261006 21126 raft_consensus.cc:2258] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:12.261473 21126 ts_tablet_manager.cc:1905] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:12.263372 21126 ts_tablet_manager.cc:1918] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:12.263497 21126 log.cc:1196] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/6af0796d5e254956bc3ae4087a524698
I20260928 21:53:12.263800 21126 ts_tablet_manager.cc:1939] T 6af0796d5e254956bc3ae4087a524698 P 9e5a3332088146f0abd880571a549dd7: Deleting consensus metadata
I20260928 21:53:12.264254 20627 catalog_manager.cc:4841] TS 9e5a3332088146f0abd880571a549dd7 (127.3.0.194:37603): tablet 6af0796d5e254956bc3ae4087a524698 (table test-table [id=61dfa76881484496ab85c6659bb77441]) successfully deleted
I20260928 21:53:12.264585 21128 tablet_replica.cc:330] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: stopping tablet replica
I20260928 21:53:12.264704 21128 raft_consensus.cc:2229] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:12.264819 21128 raft_consensus.cc:2258] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 21:53:12.264863 20624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:42445 (error 108)
I20260928 21:53:12.265239 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:12.265137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:42445 (local address 127.0.0.1:47476)
0928 21:53:12.265174 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:12.265209 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:42445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 21:53:12.265393 20624 catalog_manager.cc:4584] TS ead5cebffee743d68bc81395c2bd3d02 (127.3.0.193:42445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6af0796d5e254956bc3ae4087a524698: Network error: Client connection negotiation failed: client connection to 127.3.0.193:42445: connect: Connection refused (error 111)
I20260928 21:53:12.265456 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 20831
I20260928 21:53:12.265347 21128 ts_tablet_manager.cc:1905] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:12.267907 21128 ts_tablet_manager.cc:1918] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:12.267974 21128 log.cc:1196] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/6af0796d5e254956bc3ae4087a524698
I20260928 21:53:12.268213 21128 ts_tablet_manager.cc:1939] T 6af0796d5e254956bc3ae4087a524698 P d274e7de5ead4b23bad53035c811b4b7: Deleting consensus metadata
I20260928 21:53:12.268857 20626 catalog_manager.cc:4841] TS d274e7de5ead4b23bad53035c811b4b7 (127.3.0.195:43415): tablet 6af0796d5e254956bc3ae4087a524698 (table test-table [id=61dfa76881484496ab85c6659bb77441]) successfully deleted
I20260928 21:53:12.272322 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 20962
I20260928 21:53:12.277891 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 20609
2026-09-28T21:53:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (686 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:12 dist-test-slave-jrck krb5kdc[21132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:12 dist-test-slave-jrck krb5kdc[21132](info): set up 2 sockets
Sep 28 21:53:12 dist-test-slave-jrck krb5kdc[21132](info): commencing operation
krb5kdc: starting...
W20260928 21:53:13.215956 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:13 dist-test-slave-jrck krb5kdc[21132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:13.325408 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:42425
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:42425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:13.405287 21145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:13.405496 21145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:13.405517 21145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:13.406642 21145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:13.406677 21145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:13.406690 21145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:13.406701 21145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:13.406716 21145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:13.408108 21145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:42425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:42425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:13.408296 21145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:13.408509 21145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:13.412854 21150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:13.412873 21151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:13.412870 21153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:13.412947 21145 server_base.cc:1034] running on GCE node
I20260928 21:53:13.413241 21145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:13.413478 21145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:13.414614 21145 hybrid_clock.cc:648] HybridClock initialized: now 1790632393414601 us; error 27 us; skew 500 ppm
Sep 28 21:53:13 dist-test-slave-jrck krb5kdc[21132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632393, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:13.416960 21145 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:13.417354 21145 webserver.cc:460] Webserver started at http://127.3.0.254:41589/ using document root <none> and password file <none>
I20260928 21:53:13.417537 21145 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:13.417583 21145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:13.417701 21145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:13.418704 21145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "6280c4b146ee42a694c0047e0eb8d06b"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "dd6c50f092e3927130fcdd9ce7f678fb"
server_key_iv: "e8743f6412043a00426b736e4392c11e"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.419034 21145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "6280c4b146ee42a694c0047e0eb8d06b"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "dd6c50f092e3927130fcdd9ce7f678fb"
server_key_iv: "e8743f6412043a00426b736e4392c11e"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.420166 21145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:53:13.420792 21160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.420912 21145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.420998 21145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "6280c4b146ee42a694c0047e0eb8d06b"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "dd6c50f092e3927130fcdd9ce7f678fb"
server_key_iv: "e8743f6412043a00426b736e4392c11e"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.421057 21145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:13.429840 21145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:13.431787 21145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:13.431900 21145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:13.435638 21145 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:42425
I20260928 21:53:13.435676 21212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:42425 every 8 connection(s)
I20260928 21:53:13.435959 21145 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:13.436753 21213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:13.438876 21213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b: Bootstrap starting.
I20260928 21:53:13.439409 21213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:13.439414 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 21145
I20260928 21:53:13.439524 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:13.439689 3075 external_mini_cluster.cc:1428] Setting key f7467adab8c9b85b1ad6f7b6cddc52d1
I20260928 21:53:13.439672 21213 log.cc:824] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:13.440601 21213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b: No bootstrap required, opened a new log
I20260928 21:53:13.442670 21213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6280c4b146ee42a694c0047e0eb8d06b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42425 } }
I20260928 21:53:13.442843 21213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:13.442938 21213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6280c4b146ee42a694c0047e0eb8d06b, State: Initialized, Role: FOLLOWER
I20260928 21:53:13.443073 21213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6280c4b146ee42a694c0047e0eb8d06b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42425 } }
I20260928 21:53:13.443150 21213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:13.443185 21213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:13.443226 21213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:13.443823 21213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6280c4b146ee42a694c0047e0eb8d06b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42425 } }
I20260928 21:53:13.443917 21213 leader_election.cc:302] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6280c4b146ee42a694c0047e0eb8d06b; no voters:
I20260928 21:53:13.444075 21213 leader_election.cc:288] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:13.444113 21218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:13.444278 21218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [term 1 LEADER]: Becoming Leader. State: Replica: 6280c4b146ee42a694c0047e0eb8d06b, State: Running, Role: LEADER
I20260928 21:53:13.444319 21213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:13.444360 21218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6280c4b146ee42a694c0047e0eb8d06b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42425 } }
I20260928 21:53:13.444706 21218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6280c4b146ee42a694c0047e0eb8d06b. Latest consensus state: current_term: 1 leader_uuid: "6280c4b146ee42a694c0047e0eb8d06b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6280c4b146ee42a694c0047e0eb8d06b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42425 } } }
I20260928 21:53:13.444717 21220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6280c4b146ee42a694c0047e0eb8d06b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6280c4b146ee42a694c0047e0eb8d06b" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42425 } } }
I20260928 21:53:13.444792 21218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:13.444811 21220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:13.444983 21225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:13.445473 21225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:13.446872 21225 catalog_manager.cc:1347] Generated new cluster ID: bb5c3d8f5a6b4a01ac2599583198e093
I20260928 21:53:13.446925 21225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:53:13 dist-test-slave-jrck krb5kdc[21132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:13.454205 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.440119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.3.0.254:42425)
0928 21:53:13.440313 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:13.440320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:13.440344 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:53:13.440849 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:13.440856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:13.440904 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:13.441018 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:13.441753 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.442598 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.443612 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.444065 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.444162 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.450808 (+ 6646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:13.450813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:13.450813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:13.450825 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:13.452680 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:13.453069 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:13.453071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:13.453072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:13.453165 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:13.453348 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:13.453349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:13.453349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:13.453617 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:13.453750 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:13.453858 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 21:53:13.453999 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260928 21:53:13.462962 21225 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:13.463449 21225 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:13.466096 21225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b: Generated new TSK 0
I20260928 21:53:13.466238 21225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:53:13.494879 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:42425
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:13.574862 21239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:13.575062 21239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:13.575084 21239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:13.576187 21239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:13.576220 21239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:13.576244 21239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:53:13.577688 21239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:42425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:13.577893 21239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:13.578071 21239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:13.582777 21247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:13.582770 21244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:13.582849 21239 server_base.cc:1034] running on GCE node
W20260928 21:53:13.582787 21245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:13.583200 21239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:13.583437 21239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:13.584545 21239 hybrid_clock.cc:648] HybridClock initialized: now 1790632393584531 us; error 25 us; skew 500 ppm
Sep 28 21:53:13 dist-test-slave-jrck krb5kdc[21132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632393, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:13.586794 21239 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:53:13.587194 21239 webserver.cc:460] Webserver started at http://127.3.0.193:42519/ using document root <none> and password file <none>
I20260928 21:53:13.587384 21239 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:13.587456 21239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:13.587575 21239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:13.588608 21239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "48454c78970442079b03b9f5fecd0c22"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "900afb5959325411b629f92f11b4c74e"
server_key_iv: "a59cab663a7943f0578d437995e30e33"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.588940 21239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "48454c78970442079b03b9f5fecd0c22"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "900afb5959325411b629f92f11b4c74e"
server_key_iv: "a59cab663a7943f0578d437995e30e33"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.590111 21239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:13.590809 21254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.590965 21239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:13.591042 21239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "48454c78970442079b03b9f5fecd0c22"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "900afb5959325411b629f92f11b4c74e"
server_key_iv: "a59cab663a7943f0578d437995e30e33"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.591097 21239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:13.607390 21239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:13.609210 21239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:13.609345 21239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:13.609571 21239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:13.609903 21239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:13.609938 21239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.609958 21239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:13.609969 21239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.615406 21239 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:34007
I20260928 21:53:13.615459 21367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:34007 every 8 connection(s)
I20260928 21:53:13.615758 21239 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:53:13.619282 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 21239
I20260928 21:53:13.619369 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:53:13.619558 3075 external_mini_cluster.cc:1428] Setting key ba20d17373187e3b9c03d3053b9eed64
Sep 28 21:53:13 dist-test-slave-jrck krb5kdc[21132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632393, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:13.625029 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.616500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:49247 (local address 127.3.0.254:42425)
0928 21:53:13.616552 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:13.616554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:13.616722 (+ 168us) server_negotiation.cc:407] Connection header received
0928 21:53:13.617191 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:13.617192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:13.617200 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:13.617255 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:13.618162 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.618657 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.619723 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.620016 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.620085 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.622274 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:13.622276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:13.622276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:13.622283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:13.623698 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:13.624053 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:13.624054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:13.624055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:13.624115 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:13.624339 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:13.624339 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:13.624340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:13.624502 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:13.624566 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:13.624834 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 21:53:13.624932 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:53:13.625094 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.616424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:42425 (local address 127.3.0.193:49247)
0928 21:53:13.616600 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:13.616642 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:13.617118 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:13.617311 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:13.617314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:13.617868 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:13.618075 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.618079 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.618725 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.618725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.619646 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.619650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.620105 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.620106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.620252 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.620767 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:13.620776 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:13.622181 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:13.623768 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:13.623774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:13.623776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:13.623985 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:13.624174 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:13.624175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:13.624175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:13.624279 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:13.624567 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:13.624572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:13.624706 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 21:53:13.624799 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:53:13.624921 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260928 21:53:13.625844 21368 heartbeater.cc:347] Connected to a master server at 127.3.0.254:42425
I20260928 21:53:13.625926 21368 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:13.626078 21368 heartbeater.cc:511] Master 127.3.0.254:42425 requested a full tablet report, sending...
I20260928 21:53:13.626597 21177 ts_manager.cc:194] Registered new tserver with Master: 48454c78970442079b03b9f5fecd0c22 (127.3.0.193:34007)
I20260928 21:53:13.627488 21177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:49247
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:53:13.641629 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:42425
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:13.720289 21373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:13.720536 21373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:13.720570 21373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:13.721758 21373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:13.721802 21373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:13.721839 21373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:53:13.723341 21373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:42425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:13.723596 21373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:13.723827 21373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:13.728614 21379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:13.728590 21381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:13.728605 21378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:13.728881 21373 server_base.cc:1034] running on GCE node
I20260928 21:53:13.729074 21373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:13.729290 21373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:13.730461 21373 hybrid_clock.cc:648] HybridClock initialized: now 1790632393730443 us; error 28 us; skew 500 ppm
Sep 28 21:53:13 dist-test-slave-jrck krb5kdc[21132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632393, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:13.732908 21373 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:53:13.733361 21373 webserver.cc:460] Webserver started at http://127.3.0.194:36227/ using document root <none> and password file <none>
I20260928 21:53:13.733578 21373 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:13.733626 21373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:13.733750 21373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:13.734771 21373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "23cfd643e7a04cac8e29378095b8e97d"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "55d6fabbce2d03b9e6f4436112b36ffe"
server_key_iv: "12bd1f09e8ed9fedda437234a59db65e"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.735116 21373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "23cfd643e7a04cac8e29378095b8e97d"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "55d6fabbce2d03b9e6f4436112b36ffe"
server_key_iv: "12bd1f09e8ed9fedda437234a59db65e"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.736370 21373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:53:13.737020 21388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.737165 21373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.737242 21373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "23cfd643e7a04cac8e29378095b8e97d"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "55d6fabbce2d03b9e6f4436112b36ffe"
server_key_iv: "12bd1f09e8ed9fedda437234a59db65e"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.737309 21373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:13.748400 21373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:13.750043 21373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:13.750174 21373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:13.750378 21373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:13.750656 21373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:13.750687 21373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.750717 21373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:13.750737 21373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.756134 21373 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:46091
I20260928 21:53:13.756172 21501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:46091 every 8 connection(s)
I20260928 21:53:13.756501 21373 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:53:13 dist-test-slave-jrck krb5kdc[21132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632393, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:13.764847 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.757170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:36701 (local address 127.3.0.254:42425)
0928 21:53:13.757224 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:13.757226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:13.757407 (+ 181us) server_negotiation.cc:407] Connection header received
0928 21:53:13.757867 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:13.757868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:13.757877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:13.757933 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:13.758817 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.759323 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.760283 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.760507 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.760559 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.762612 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:13.762614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:13.762614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:13.762620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:13.763687 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:13.764058 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:13.764059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:13.764059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:13.764102 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:13.764277 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:13.764278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:13.764278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:13.764372 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:13.764416 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:13.764717 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 21:53:13.764780 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:53:13.765043 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.757141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:42425 (local address 127.3.0.194:36701)
0928 21:53:13.757282 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:13.757323 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:13.757795 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:13.757986 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:13.757990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:13.758543 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:13.758738 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.758743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.759405 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.759406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.760219 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.760221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.760585 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.760586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.760705 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.761158 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:13.761166 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:13.762527 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:13.763766 (+ 1239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:13.763773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:13.763774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:13.763996 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:13.764148 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:13.764148 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:13.764148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:13.764237 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:13.764444 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:13.764451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:13.764597 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 21:53:13.764685 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:53:13.764812 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260928 21:53:13.765499 21502 heartbeater.cc:347] Connected to a master server at 127.3.0.254:42425
I20260928 21:53:13.765585 21502 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:13.765751 21502 heartbeater.cc:511] Master 127.3.0.254:42425 requested a full tablet report, sending...
I20260928 21:53:13.765936 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 21373
I20260928 21:53:13.766052 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:53:13.766146 21177 ts_manager.cc:194] Registered new tserver with Master: 23cfd643e7a04cac8e29378095b8e97d (127.3.0.194:46091)
I20260928 21:53:13.766218 3075 external_mini_cluster.cc:1428] Setting key 7ffcd091e4072993ccde694b389945d4
I20260928 21:53:13.766520 21177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:36701
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:53:13.788242 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:42425
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:13.866981 21507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:13.867173 21507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:13.867194 21507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:13.868270 21507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:13.868311 21507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:13.868335 21507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:53:13.869731 21507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:42425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:13.869912 21507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:13.870082 21507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:13.874621 21515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:13.874754 21507 server_base.cc:1034] running on GCE node
W20260928 21:53:13.874621 21512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:13.874698 21513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:13.875028 21507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:13.875257 21507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:13.876398 21507 hybrid_clock.cc:648] HybridClock initialized: now 1790632393876385 us; error 28 us; skew 500 ppm
Sep 28 21:53:13 dist-test-slave-jrck krb5kdc[21132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632393, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:13.878769 21507 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:53:13.879168 21507 webserver.cc:460] Webserver started at http://127.3.0.195:44153/ using document root <none> and password file <none>
I20260928 21:53:13.879364 21507 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:13.879410 21507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:13.879532 21507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:13.880554 21507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "377cf52c02af4b71946d70a083f81dd5"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "8a51612a3bb13695bcc97a9dee81a0bd"
server_key_iv: "4193fafc0b8f2b837e2415ad9de90c62"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.880867 21507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "377cf52c02af4b71946d70a083f81dd5"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "8a51612a3bb13695bcc97a9dee81a0bd"
server_key_iv: "4193fafc0b8f2b837e2415ad9de90c62"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.882033 21507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:53:13.882668 21522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.882803 21507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:53:13.882879 21507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "377cf52c02af4b71946d70a083f81dd5"
format_stamp: "Formatted at 2026-09-28 21:53:13 on dist-test-slave-jrck"
server_key: "8a51612a3bb13695bcc97a9dee81a0bd"
server_key_iv: "4193fafc0b8f2b837e2415ad9de90c62"
server_key_version: "encryptionkey@0"
I20260928 21:53:13.882934 21507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:13.900030 21507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:13.901830 21507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:13.901983 21507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:13.902187 21507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:13.902474 21507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:13.902511 21507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.902540 21507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:13.902562 21507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:13.907801 21507 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:43765
I20260928 21:53:13.907843 21635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:43765 every 8 connection(s)
I20260928 21:53:13.908143 21507 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:53:13.912704 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 21507
I20260928 21:53:13.912783 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:53:13.912949 3075 external_mini_cluster.cc:1428] Setting key a07b4b00119b1cbf96e350b7c4ab8a97
Sep 28 21:53:13 dist-test-slave-jrck krb5kdc[21132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632393, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:13.916209 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.908801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:49663 (local address 127.3.0.254:42425)
0928 21:53:13.908911 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:13.908914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:13.909021 (+ 107us) server_negotiation.cc:407] Connection header received
0928 21:53:13.909470 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:13.909471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:13.909481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:13.909527 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:13.910374 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.910867 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.911792 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.911999 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.912051 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.914139 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:13.914140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:13.914140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:13.914147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:13.915152 (+ 1005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:13.915544 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:13.915545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:13.915545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:13.915585 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:13.915757 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:13.915757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:13.915757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:13.915846 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:13.915889 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:13.916093 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 21:53:13.916146 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 21:53:13.916388 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.908770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:42425 (local address 127.3.0.195:49663)
0928 21:53:13.908908 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:13.908942 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:13.909393 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:13.909580 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:13.909584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:13.910120 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:13.910303 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.910308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.910928 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.910929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.911717 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.911720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.912062 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.912063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.912157 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.912600 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:13.912608 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:13.914067 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:13.915264 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:13.915269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:13.915271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:13.915483 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:13.915628 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:13.915628 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:13.915629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:13.915716 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:13.915887 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:13.915890 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:13.915999 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 21:53:13.916075 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 21:53:13.916186 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260928 21:53:13.916797 21636 heartbeater.cc:347] Connected to a master server at 127.3.0.254:42425
I20260928 21:53:13.916913 21636 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:13.917078 21636 heartbeater.cc:511] Master 127.3.0.254:42425 requested a full tablet report, sending...
I20260928 21:53:13.917398 21176 ts_manager.cc:194] Registered new tserver with Master: 377cf52c02af4b71946d70a083f81dd5 (127.3.0.195:43765)
I20260928 21:53:13.917757 21176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:49663
I20260928 21:53:13.924571 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:53:13.931481 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.925907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.3.0.254:42425)
0928 21:53:13.925942 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:13.925943 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:13.926076 (+ 133us) server_negotiation.cc:407] Connection header received
0928 21:53:13.926136 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:13.926137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:13.926146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:13.926194 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:13.926712 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.927188 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.927887 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.928087 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.928139 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.929353 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:13.929354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:13.929355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:13.929362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:13.930382 (+ 1020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:13.930737 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:13.930739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:13.930739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:13.930793 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:13.930981 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:13.930982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:13.930982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:13.931118 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:13.931213 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:13.931323 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 21:53:13.931381 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:53:13.932801 21176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:53:13.934374 21176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:53:13.940286 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.936221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:46091 (local address 127.0.0.1:52376)
0928 21:53:13.936575 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:13.936617 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:13.936683 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:13.937054 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:13.937055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:13.937057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:13.937102 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.937103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.938453 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.938454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.939299 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.939301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.940053 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.940054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.940104 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.940107 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:13.940162 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:53:13.940216 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":54,"threads_started":1}
I20260928 21:53:13.940299 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.936367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.3.0.193:34007)
0928 21:53:13.936571 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:13.936576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:13.936797 (+ 221us) server_negotiation.cc:407] Connection header received
0928 21:53:13.936854 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:13.936856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:13.936869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:13.936942 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:13.937133 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.937939 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.939142 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.939862 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.939896 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.939937 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:13.940172 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 21:53:13.940210 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":109,"threads_started":1}
I20260928 21:53:13.940296 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.936588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52376 (local address 127.3.0.194:46091)
0928 21:53:13.936891 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:13.936894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:13.936901 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:53:13.936929 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:13.936930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:13.936941 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:13.937006 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:13.937246 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.938342 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.939374 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.939975 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.940044 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.940097 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:13.940177 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 21:53:13.940217 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":71,"threads_started":1}
I20260928 21:53:13.940515 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.936235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:34007 (local address 127.0.0.1:48896)
0928 21:53:13.936742 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:13.936751 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:13.936799 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:13.936970 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:13.936973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:13.936976 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:13.937047 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.937049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.938043 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.938044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.939047 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.939050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.939989 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.939989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.940054 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.940068 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 21:53:13.940397 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 21:53:13.940433 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":49,"threads_started":1}
I20260928 21:53:13.940903 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.936420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:43765 (local address 127.0.0.1:55318)
0928 21:53:13.937096 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:13.937105 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:13.937148 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:13.937339 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:13.937340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:13.937342 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:13.937374 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.937376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.938544 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.938545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.939616 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:13.939619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.940676 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.940677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:13.940737 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.940740 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:13.940786 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:53:13.940825 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":87,"threads_started":1}
I20260928 21:53:13.941135 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:13.936555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.3.0.195:43765)
0928 21:53:13.936716 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:13.936720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:13.937204 (+ 484us) server_negotiation.cc:407] Connection header received
0928 21:53:13.937253 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:13.937256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:13.937270 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:13.937342 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:13.937421 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.938451 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.939694 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:13.940527 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:13.940593 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:13.940645 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:13.941010 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 21:53:13.941059 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260928 21:53:13.941257 21302 tablet_service.cc:1467] Processing CreateTablet for tablet 45f6e63cb68f49f992430a260fd39ed5 (DEFAULT_TABLE table=test-table [id=385d9c24ed914d0c82bbb70fab77f5de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:13.941613 21302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f6e63cb68f49f992430a260fd39ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:13.941720 21570 tablet_service.cc:1467] Processing CreateTablet for tablet 45f6e63cb68f49f992430a260fd39ed5 (DEFAULT_TABLE table=test-table [id=385d9c24ed914d0c82bbb70fab77f5de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:13.941998 21570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f6e63cb68f49f992430a260fd39ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:13.944135 21650 tablet_bootstrap.cc:492] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22: Bootstrap starting.
I20260928 21:53:13.944589 21651 tablet_bootstrap.cc:492] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: Bootstrap starting.
I20260928 21:53:13.944512 21436 tablet_service.cc:1467] Processing CreateTablet for tablet 45f6e63cb68f49f992430a260fd39ed5 (DEFAULT_TABLE table=test-table [id=385d9c24ed914d0c82bbb70fab77f5de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:13.944811 21436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45f6e63cb68f49f992430a260fd39ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:13.945029 21650 tablet_bootstrap.cc:654] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:13.945191 21651 tablet_bootstrap.cc:654] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:13.945597 21650 log.cc:824] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:13.945709 21651 log.cc:824] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:13.946297 21650 tablet_bootstrap.cc:492] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22: No bootstrap required, opened a new log
I20260928 21:53:13.946359 21651 tablet_bootstrap.cc:492] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: No bootstrap required, opened a new log
I20260928 21:53:13.946379 21650 ts_tablet_manager.cc:1397] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:53:13.946413 21651 ts_tablet_manager.cc:1397] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:53:13.947496 21654 tablet_bootstrap.cc:492] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: Bootstrap starting.
I20260928 21:53:13.947980 21654 tablet_bootstrap.cc:654] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:13.948226 21651 raft_consensus.cc:348] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23cfd643e7a04cac8e29378095b8e97d" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46091 } } peers { permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 } } peers { permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 } }
I20260928 21:53:13.948226 21650 raft_consensus.cc:348] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23cfd643e7a04cac8e29378095b8e97d" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46091 } } peers { permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 } } peers { permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 } }
I20260928 21:53:13.948408 21651 raft_consensus.cc:374] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:13.948408 21650 raft_consensus.cc:374] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:13.948416 21654 log.cc:824] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:13.948473 21650 raft_consensus.cc:729] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48454c78970442079b03b9f5fecd0c22, State: Initialized, Role: FOLLOWER
I20260928 21:53:13.948473 21651 raft_consensus.cc:729] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 377cf52c02af4b71946d70a083f81dd5, State: Initialized, Role: FOLLOWER
I20260928 21:53:13.948590 21650 consensus_queue.cc:260] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23cfd643e7a04cac8e29378095b8e97d" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46091 } } peers { permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 } } peers { permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 } }
I20260928 21:53:13.948642 21651 consensus_queue.cc:260] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23cfd643e7a04cac8e29378095b8e97d" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46091 } } peers { permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 } } peers { permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 } }
I20260928 21:53:13.948884 21651 ts_tablet_manager.cc:1428] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:53:13.948884 21368 heartbeater.cc:503] Master 127.3.0.254:42425 was elected leader, sending a full tablet report...
I20260928 21:53:13.948988 21636 heartbeater.cc:503] Master 127.3.0.254:42425 was elected leader, sending a full tablet report...
I20260928 21:53:13.949191 21654 tablet_bootstrap.cc:492] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: No bootstrap required, opened a new log
I20260928 21:53:13.948880 21650 ts_tablet_manager.cc:1428] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:53:13.949254 21654 ts_tablet_manager.cc:1397] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:53:13.950690 21654 raft_consensus.cc:348] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23cfd643e7a04cac8e29378095b8e97d" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46091 } } peers { permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 } } peers { permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 } }
I20260928 21:53:13.950820 21654 raft_consensus.cc:374] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:13.950850 21654 raft_consensus.cc:729] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23cfd643e7a04cac8e29378095b8e97d, State: Initialized, Role: FOLLOWER
I20260928 21:53:13.950963 21654 consensus_queue.cc:260] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23cfd643e7a04cac8e29378095b8e97d" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46091 } } peers { permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 } } peers { permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 } }
I20260928 21:53:13.951215 21654 ts_tablet_manager.cc:1428] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:53:13.951218 21502 heartbeater.cc:503] Master 127.3.0.254:42425 was elected leader, sending a full tablet report...
W20260928 21:53:14.007055 21503 tablet.cc:2369] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:53:14.116501 21369 tablet.cc:2369] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:53:14.158696 21637 tablet.cc:2369] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:53:14.166647 21658 raft_consensus.cc:482] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:53:14.166759 21658 raft_consensus.cc:504] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23cfd643e7a04cac8e29378095b8e97d" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46091 } } peers { permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 } } peers { permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 } }
I20260928 21:53:14.167095 21658 leader_election.cc:288] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48454c78970442079b03b9f5fecd0c22 (127.3.0.193:34007), 377cf52c02af4b71946d70a083f81dd5 (127.3.0.195:43765)
I20260928 21:53:14.170789 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.167341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:53099 (local address 127.3.0.193:34007)
0928 21:53:14.167406 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.167408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.167418 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:53:14.167445 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.167446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.167460 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.167512 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:14.167661 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.168405 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.169840 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.170413 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.170488 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.170498 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.170670 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.170713 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:53:14.170789 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.167594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:58127 (local address 127.3.0.195:43765)
0928 21:53:14.167646 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.167648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.167655 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:53:14.167678 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.167679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.167689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.167740 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:14.167849 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.168556 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.169865 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.170416 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.170485 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.170498 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.170670 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.170717 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:53:14.170820 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.167266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:34007 (local address 127.3.0.194:53099)
0928 21:53:14.167321 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:14.167335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:14.167384 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:14.167530 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:14.167531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:14.167533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:14.167605 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.167607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.168516 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.168517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.169725 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.169728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.170495 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.170496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.170600 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.170606 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:53:14.170656 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:53:14.170717 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":26}
I20260928 21:53:14.170826 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.167397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:43765 (local address 127.3.0.194:58127)
0928 21:53:14.167559 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:14.167568 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:14.167609 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:14.167747 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:14.167747 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:14.167748 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:14.167792 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.167792 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.168615 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.168616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.169725 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.169728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.170495 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.170496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.170600 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.170606 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:53:14.170656 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:53:14.170717 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":21,"thread_start_us":100,"threads_started":1}
I20260928 21:53:14.171238 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f6e63cb68f49f992430a260fd39ed5" candidate_uuid: "23cfd643e7a04cac8e29378095b8e97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48454c78970442079b03b9f5fecd0c22" is_pre_election: true
I20260928 21:53:14.171238 21590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f6e63cb68f49f992430a260fd39ed5" candidate_uuid: "23cfd643e7a04cac8e29378095b8e97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "377cf52c02af4b71946d70a083f81dd5" is_pre_election: true
I20260928 21:53:14.171412 21590 raft_consensus.cc:2454] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23cfd643e7a04cac8e29378095b8e97d in term 0.
I20260928 21:53:14.171412 21322 raft_consensus.cc:2454] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23cfd643e7a04cac8e29378095b8e97d in term 0.
I20260928 21:53:14.171633 21390 leader_election.cc:302] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23cfd643e7a04cac8e29378095b8e97d, 48454c78970442079b03b9f5fecd0c22; no voters:
I20260928 21:53:14.171797 21658 raft_consensus.cc:2781] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:53:14.171865 21658 raft_consensus.cc:482] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:53:14.171882 21658 raft_consensus.cc:3037] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:14.172591 21658 raft_consensus.cc:504] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23cfd643e7a04cac8e29378095b8e97d" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46091 } } peers { permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 } } peers { permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 } }
I20260928 21:53:14.172703 21658 leader_election.cc:288] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [CANDIDATE]: Term 1 election: Requested vote from peers 48454c78970442079b03b9f5fecd0c22 (127.3.0.193:34007), 377cf52c02af4b71946d70a083f81dd5 (127.3.0.195:43765)
I20260928 21:53:14.172878 21322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f6e63cb68f49f992430a260fd39ed5" candidate_uuid: "23cfd643e7a04cac8e29378095b8e97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48454c78970442079b03b9f5fecd0c22"
I20260928 21:53:14.172961 21322 raft_consensus.cc:3037] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:14.172974 21590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45f6e63cb68f49f992430a260fd39ed5" candidate_uuid: "23cfd643e7a04cac8e29378095b8e97d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "377cf52c02af4b71946d70a083f81dd5"
I20260928 21:53:14.173066 21590 raft_consensus.cc:3037] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:14.173677 21322 raft_consensus.cc:2454] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23cfd643e7a04cac8e29378095b8e97d in term 1.
I20260928 21:53:14.173748 21590 raft_consensus.cc:2454] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23cfd643e7a04cac8e29378095b8e97d in term 1.
I20260928 21:53:14.173861 21390 leader_election.cc:302] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23cfd643e7a04cac8e29378095b8e97d, 48454c78970442079b03b9f5fecd0c22; no voters:
I20260928 21:53:14.173960 21658 raft_consensus.cc:2781] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:14.174106 21658 raft_consensus.cc:686] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 1 LEADER]: Becoming Leader. State: Replica: 23cfd643e7a04cac8e29378095b8e97d, State: Running, Role: LEADER
I20260928 21:53:14.174193 21658 consensus_queue.cc:237] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23cfd643e7a04cac8e29378095b8e97d" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46091 } } peers { permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 } } peers { permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 } }
I20260928 21:53:14.175035 21177 catalog_manager.cc:5495] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d reported cstate change: term changed from 0 to 1, leader changed from <none> to 23cfd643e7a04cac8e29378095b8e97d (127.3.0.194). New cstate: current_term: 1 leader_uuid: "23cfd643e7a04cac8e29378095b8e97d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23cfd643e7a04cac8e29378095b8e97d" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 } health_report { overall_health: UNKNOWN } } }
I20260928 21:53:14.226661 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.224329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.3.0.194:46091)
0928 21:53:14.224382 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.224384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.224392 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:53:14.224429 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.224430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.224472 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.224538 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:14.224670 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.225174 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.225932 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.226132 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.226187 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.226344 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:14.226438 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:14.226522 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.226569 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.226576 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:53:14.228410 21590 raft_consensus.cc:1261] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5 [term 1 FOLLOWER]: Refusing update from remote peer 23cfd643e7a04cac8e29378095b8e97d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:14.228520 21322 raft_consensus.cc:1261] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22 [term 1 FOLLOWER]: Refusing update from remote peer 23cfd643e7a04cac8e29378095b8e97d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:14.228792 21660 consensus_queue.cc:1035] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [LEADER]: Connected to new peer: Peer: permanent_uuid: "377cf52c02af4b71946d70a083f81dd5" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:14.228928 21658 consensus_queue.cc:1035] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [LEADER]: Connected to new peer: Peer: permanent_uuid: "48454c78970442079b03b9f5fecd0c22" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 34007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:14.233381 21177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:14.233477 21177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:14.234230 21177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6280c4b146ee42a694c0047e0eb8d06b: Sending DeleteTablet for 3 replicas of tablet 45f6e63cb68f49f992430a260fd39ed5
I20260928 21:53:14.234511 21302 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f6e63cb68f49f992430a260fd39ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:14 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:48896
I20260928 21:53:14.234529 21301 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f6e63cb68f49f992430a260fd39ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:14 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:48896
I20260928 21:53:14.234592 21570 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f6e63cb68f49f992430a260fd39ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:14 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:55318
I20260928 21:53:14.234727 21672 tablet_replica.cc:330] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22: stopping tablet replica
I20260928 21:53:14.234830 21672 raft_consensus.cc:2229] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:14.234956 21672 raft_consensus.cc:2258] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:14.235037 21436 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f6e63cb68f49f992430a260fd39ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:14 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:52376
I20260928 21:53:14.235236 21674 tablet_replica.cc:330] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: stopping tablet replica
I20260928 21:53:14.235363 21674 raft_consensus.cc:2229] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:53:14.235400 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 21239
I20260928 21:53:14.235421 21672 ts_tablet_manager.cc:1905] T 45f6e63cb68f49f992430a260fd39ed5 P 48454c78970442079b03b9f5fecd0c22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:14.235496 21674 raft_consensus.cc:2258] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:14.237110 21674 ts_tablet_manager.cc:1905] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:14.238729 21674 ts_tablet_manager.cc:1918] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:14.238849 21674 log.cc:1196] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/45f6e63cb68f49f992430a260fd39ed5
I20260928 21:53:14.239132 21674 ts_tablet_manager.cc:1939] T 45f6e63cb68f49f992430a260fd39ed5 P 23cfd643e7a04cac8e29378095b8e97d: Deleting consensus metadata
I20260928 21:53:14.239683 21164 catalog_manager.cc:4841] TS 23cfd643e7a04cac8e29378095b8e97d (127.3.0.194:46091): tablet 45f6e63cb68f49f992430a260fd39ed5 (table test-table [id=385d9c24ed914d0c82bbb70fab77f5de]) successfully deleted
W20260928 21:53:14.242208 21162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:34007 (error 108)
I20260928 21:53:14.242282 21673 tablet_replica.cc:330] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: stopping tablet replica
I20260928 21:53:14.242435 21673 raft_consensus.cc:2229] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:14.242558 21673 raft_consensus.cc:2258] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:14.242712 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.242574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:34007 (local address 127.0.0.1:48898)
0928 21:53:14.242629 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:14.242677 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 21:53:14.242760 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 21373
W20260928 21:53:14.242830 21162 catalog_manager.cc:4584] TS 48454c78970442079b03b9f5fecd0c22 (127.3.0.193:34007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45f6e63cb68f49f992430a260fd39ed5: Network error: Client connection negotiation failed: client connection to 127.3.0.193:34007: connect: Connection refused (error 111)
I20260928 21:53:14.245095 21673 ts_tablet_manager.cc:1905] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:14.246501 21673 ts_tablet_manager.cc:1918] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:14.246569 21673 log.cc:1196] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/45f6e63cb68f49f992430a260fd39ed5
I20260928 21:53:14.246773 21673 ts_tablet_manager.cc:1939] T 45f6e63cb68f49f992430a260fd39ed5 P 377cf52c02af4b71946d70a083f81dd5: Deleting consensus metadata
I20260928 21:53:14.247417 21162 catalog_manager.cc:4841] TS 377cf52c02af4b71946d70a083f81dd5 (127.3.0.195:43765): tablet 45f6e63cb68f49f992430a260fd39ed5 (table test-table [id=385d9c24ed914d0c82bbb70fab77f5de]) successfully deleted
I20260928 21:53:14.247828 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 21507
I20260928 21:53:14.252324 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 21145
2026-09-28T21:53:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1972 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T21:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:14Z Disabled control of system clock
I20260928 21:53:14.277606 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:37259
--webserver_interface=127.3.0.254
--webserver_port=0
--builtin_ntp_servers=127.3.0.212:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:37259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:14.354712 21679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:14.354929 21679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:14.354954 21679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:14.356040 21679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:14.356071 21679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:14.356082 21679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:14.356092 21679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:14.356102 21679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:14.357384 21679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:37259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:37259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:14.357573 21679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:14.357795 21679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:14.362326 21684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:14.362437 21679 server_base.cc:1034] running on GCE node
W20260928 21:53:14.362316 21687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:14.362320 21685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:14.362772 21679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:14.363046 21679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:14.364180 21679 hybrid_clock.cc:648] HybridClock initialized: now 1790632394364166 us; error 29 us; skew 500 ppm
I20260928 21:53:14.365314 21679 webserver.cc:460] Webserver started at http://127.3.0.254:45445/ using document root <none> and password file <none>
I20260928 21:53:14.365517 21679 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:14.365566 21679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:14.365689 21679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:14.366694 21679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "0fa64d37c602471a8b08011370a64273"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "58a5aaf6648980707c6c9ce64700a1d8"
server_key_iv: "f5af9fa828102d41e3b7884bd3e46f78"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.367015 21679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "0fa64d37c602471a8b08011370a64273"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "58a5aaf6648980707c6c9ce64700a1d8"
server_key_iv: "f5af9fa828102d41e3b7884bd3e46f78"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.368122 21679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:14.368762 21693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.368898 21679 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.368990 21679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "0fa64d37c602471a8b08011370a64273"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "58a5aaf6648980707c6c9ce64700a1d8"
server_key_iv: "f5af9fa828102d41e3b7884bd3e46f78"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.369047 21679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:14.379304 21679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:14.379599 21679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:14.379725 21679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:14.383488 21679 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:37259
I20260928 21:53:14.383518 21745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:37259 every 8 connection(s)
I20260928 21:53:14.383831 21679 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:14.384757 21746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:14.387019 21746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273: Bootstrap starting.
I20260928 21:53:14.387557 21746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:14.387818 21746 log.cc:824] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:14.388443 21746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273: No bootstrap required, opened a new log
I20260928 21:53:14.390038 21746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa64d37c602471a8b08011370a64273" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37259 } }
I20260928 21:53:14.390172 21746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:14.390202 21746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa64d37c602471a8b08011370a64273, State: Initialized, Role: FOLLOWER
I20260928 21:53:14.390342 21746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa64d37c602471a8b08011370a64273" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37259 } }
I20260928 21:53:14.390417 21746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:14.390450 21746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:14.390496 21746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:14.390981 21746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa64d37c602471a8b08011370a64273" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37259 } }
I20260928 21:53:14.391070 21746 leader_election.cc:302] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fa64d37c602471a8b08011370a64273; no voters:
I20260928 21:53:14.391224 21746 leader_election.cc:288] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:14.391243 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 21679
I20260928 21:53:14.391266 21749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:14.391326 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:14.391429 21749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [term 1 LEADER]: Becoming Leader. State: Replica: 0fa64d37c602471a8b08011370a64273, State: Running, Role: LEADER
I20260928 21:53:14.391477 21746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:14.391520 3075 external_mini_cluster.cc:1428] Setting key 728f80dc4ea3aa5a5646b6cc6d2a8bf2
I20260928 21:53:14.391651 21749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa64d37c602471a8b08011370a64273" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37259 } }
I20260928 21:53:14.392069 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fa64d37c602471a8b08011370a64273. Latest consensus state: current_term: 1 leader_uuid: "0fa64d37c602471a8b08011370a64273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa64d37c602471a8b08011370a64273" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37259 } } }
I20260928 21:53:14.392139 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fa64d37c602471a8b08011370a64273" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa64d37c602471a8b08011370a64273" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37259 } } }
I20260928 21:53:14.392230 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:14.392217 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:14.392555 21760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:14.393072 21760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:14.394651 21760 catalog_manager.cc:1347] Generated new cluster ID: 4ef417d01f9f4aefbe01fc900b0d5a63
I20260928 21:53:14.394701 21760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:14.397727 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.392002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.3.0.254:37259)
0928 21:53:14.392217 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.392227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.392407 (+ 180us) server_negotiation.cc:407] Connection header received
0928 21:53:14.392551 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.392559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.392632 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.392764 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:14.393201 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.394363 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.395460 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.396058 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.396171 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.396913 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:14.396946 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:14.396946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:14.397145 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:14.397185 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:14.397199 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:14.397257 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.397344 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.397520 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260928 21:53:14.405783 21760 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:14.406512 21760 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:14.409219 21760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273: Generated new TSK 0
I20260928 21:53:14.409386 21760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:14.413501 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:37259
--builtin_ntp_servers=127.3.0.212:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:14.493458 21770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:14.493681 21770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:14.493714 21770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:14.494910 21770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:14.494956 21770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:14.494992 21770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:53:14.496361 21770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:14.496614 21770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:14.496891 21770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:53:14.501719 21770 server_base.cc:1034] running on GCE node
W20260928 21:53:14.501693 21776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:14.501824 21778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:14.501688 21775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:14.502110 21770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:14.502305 21770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:14.503437 21770 hybrid_clock.cc:648] HybridClock initialized: now 1790632394503418 us; error 35 us; skew 500 ppm
I20260928 21:53:14.504691 21770 webserver.cc:460] Webserver started at http://127.3.0.193:43929/ using document root <none> and password file <none>
I20260928 21:53:14.504908 21770 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:14.504956 21770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:14.505095 21770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:14.506121 21770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "b0e25e9db70e4f1795ad3adae17f46f4"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "245236a6f5b1f625657080ab4cdeb918"
server_key_iv: "f1f74e0c9d2bbaa4a558825a75cc30c9"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.506443 21770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "b0e25e9db70e4f1795ad3adae17f46f4"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "245236a6f5b1f625657080ab4cdeb918"
server_key_iv: "f1f74e0c9d2bbaa4a558825a75cc30c9"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.507685 21770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:14.508366 21784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.508543 21770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:53:14.508615 21770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "b0e25e9db70e4f1795ad3adae17f46f4"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "245236a6f5b1f625657080ab4cdeb918"
server_key_iv: "f1f74e0c9d2bbaa4a558825a75cc30c9"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.508687 21770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:14.522768 21770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:14.523072 21770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:14.523191 21770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:14.523414 21770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:14.523715 21770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:14.523749 21770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.523770 21770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:14.523782 21770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.529361 21770 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:38581
I20260928 21:53:14.529412 21897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:38581 every 8 connection(s)
I20260928 21:53:14.529716 21770 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:53:14.534765 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.530382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:54063 (local address 127.3.0.254:37259)
0928 21:53:14.530434 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.530437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.530704 (+ 267us) server_negotiation.cc:407] Connection header received
0928 21:53:14.531233 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.531234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.531247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.531288 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:14.532037 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.532581 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.533525 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.533718 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.533771 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.534547 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:14.534548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:14.534549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:14.534552 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:14.534566 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:14.534570 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:14.534615 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.534675 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.534704 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:53:14.534992 21900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.530373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:37259 (local address 127.3.0.193:54063)
0928 21:53:14.530577 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:14.530612 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:14.531137 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:14.531399 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:14.531403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:14.531414 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:14.531963 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.531968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.532662 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.532663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.533443 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.533445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.533802 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.533803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.533918 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.534407 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:53:14.534408 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:14.534436 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:53:14.534459 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:14.534619 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:14.534622 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:14.534624 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:53:14.534671 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:53:14.534749 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260928 21:53:14.535434 21898 heartbeater.cc:347] Connected to a master server at 127.3.0.254:37259
I20260928 21:53:14.535518 21898 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:14.535663 21898 heartbeater.cc:511] Master 127.3.0.254:37259 requested a full tablet report, sending...
I20260928 21:53:14.536157 21710 ts_manager.cc:194] Registered new tserver with Master: b0e25e9db70e4f1795ad3adae17f46f4 (127.3.0.193:38581)
I20260928 21:53:14.536899 21710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.193:54063
I20260928 21:53:14.537961 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 21770
I20260928 21:53:14.538045 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:53:14.538264 3075 external_mini_cluster.cc:1428] Setting key 0e781c8cdf9bdc0f4f5aaa8166f49332
I20260928 21:53:14.539196 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:37259
--builtin_ntp_servers=127.3.0.212:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:14.617617 21901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:14.617806 21901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:14.617826 21901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:14.618933 21901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:14.618971 21901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:14.618995 21901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:53:14.620280 21901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:14.620514 21901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:14.620790 21901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:14.625684 21906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:14.625746 21909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:14.625716 21901 server_base.cc:1034] running on GCE node
W20260928 21:53:14.625684 21907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:14.626066 21901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:14.626279 21901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:14.627444 21901 hybrid_clock.cc:648] HybridClock initialized: now 1790632394627426 us; error 30 us; skew 500 ppm
I20260928 21:53:14.628654 21901 webserver.cc:460] Webserver started at http://127.3.0.194:36709/ using document root <none> and password file <none>
I20260928 21:53:14.628921 21901 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:14.628971 21901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:14.629096 21901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:14.630129 21901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "f549ff9e37214a2bbab08d0873d49276"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "c7760b3c64b481a2018ebb244b6858f2"
server_key_iv: "c5f369f4f706bf024497d7ba383ec7ba"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.630445 21901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "f549ff9e37214a2bbab08d0873d49276"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "c7760b3c64b481a2018ebb244b6858f2"
server_key_iv: "c5f369f4f706bf024497d7ba383ec7ba"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.631660 21901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:14.632405 21915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.632592 21901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:14.632665 21901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "f549ff9e37214a2bbab08d0873d49276"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "c7760b3c64b481a2018ebb244b6858f2"
server_key_iv: "c5f369f4f706bf024497d7ba383ec7ba"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.632716 21901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:14.649952 21901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:14.650220 21901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:14.650317 21901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:14.650509 21901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:14.650789 21901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:14.650818 21901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.650838 21901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:14.650854 21901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.656080 21901 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:42927
I20260928 21:53:14.656160 22028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:42927 every 8 connection(s)
I20260928 21:53:14.656427 21901 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:53:14.661420 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.657128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:46025 (local address 127.3.0.254:37259)
0928 21:53:14.657171 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.657173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.657406 (+ 233us) server_negotiation.cc:407] Connection header received
0928 21:53:14.657878 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.657879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.657888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.657943 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:14.658553 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.659132 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.660179 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.660357 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.660413 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.661205 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:14.661206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:14.661206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:14.661210 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:14.661225 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:14.661228 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:14.661267 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.661336 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.661361 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:53:14.661626 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.657107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:37259 (local address 127.3.0.194:46025)
0928 21:53:14.657288 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:14.657329 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:14.657795 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:14.658003 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:14.658007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:14.658014 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:14.658494 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.658498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.659255 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.659256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.660092 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.660094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.660438 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.660439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.660569 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.661073 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:53:14.661074 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:14.661103 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:53:14.661128 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:14.661276 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:14.661279 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:14.661281 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:53:14.661333 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:53:14.661415 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260928 21:53:14.662060 22029 heartbeater.cc:347] Connected to a master server at 127.3.0.254:37259
I20260928 21:53:14.662141 22029 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:14.662310 22029 heartbeater.cc:511] Master 127.3.0.254:37259 requested a full tablet report, sending...
I20260928 21:53:14.662738 21710 ts_manager.cc:194] Registered new tserver with Master: f549ff9e37214a2bbab08d0873d49276 (127.3.0.194:42927)
I20260928 21:53:14.663115 21710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.194:46025
I20260928 21:53:14.663546 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 21901
I20260928 21:53:14.663628 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:53:14.663796 3075 external_mini_cluster.cc:1428] Setting key ed5c21164e9eab882ba4910e614272d8
I20260928 21:53:14.664705 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:37259
--builtin_ntp_servers=127.3.0.212:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:14.743556 22032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:14.743758 22032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:14.743778 22032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:14.744907 22032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:14.744943 22032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:14.744967 22032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:53:14.746238 22032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:37259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:14.746416 22032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:14.746651 22032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:14.751274 22040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:14.751293 22037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:14.751382 22032 server_base.cc:1034] running on GCE node
W20260928 21:53:14.751281 22038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:14.751652 22032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:14.751878 22032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:14.753054 22032 hybrid_clock.cc:648] HybridClock initialized: now 1790632394753044 us; error 26 us; skew 500 ppm
I20260928 21:53:14.754266 22032 webserver.cc:460] Webserver started at http://127.3.0.195:45463/ using document root <none> and password file <none>
I20260928 21:53:14.754485 22032 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:14.754532 22032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:14.754650 22032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:14.755699 22032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "f45adbfaed7248ad8d26d3e46ebb086b"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "6eca5ff7237ab0de2ccc35bd28e61508"
server_key_iv: "910b2f1bf9054b9609df157713adafb0"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.756062 22032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "f45adbfaed7248ad8d26d3e46ebb086b"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "6eca5ff7237ab0de2ccc35bd28e61508"
server_key_iv: "910b2f1bf9054b9609df157713adafb0"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.757272 22032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:14.757928 22046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.758059 22032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:14.758133 22032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "f45adbfaed7248ad8d26d3e46ebb086b"
format_stamp: "Formatted at 2026-09-28 21:53:14 on dist-test-slave-jrck"
server_key: "6eca5ff7237ab0de2ccc35bd28e61508"
server_key_iv: "910b2f1bf9054b9609df157713adafb0"
server_key_version: "encryptionkey@0"
I20260928 21:53:14.758195 22032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:14.771844 22032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:14.772140 22032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:14.772260 22032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:14.772541 22032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:14.772884 22032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:14.772917 22032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.772957 22032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:14.772979 22032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:14.778194 22032 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:46795
I20260928 21:53:14.778245 22159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:46795 every 8 connection(s)
I20260928 21:53:14.778549 22032 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:53:14.778781 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 22032
I20260928 21:53:14.778880 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:53:14.779071 3075 external_mini_cluster.cc:1428] Setting key 44e075dd09509af406e61f9702cc3f22
I20260928 21:53:14.783521 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.779247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:45099 (local address 127.3.0.254:37259)
0928 21:53:14.779291 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.779293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.779571 (+ 278us) server_negotiation.cc:407] Connection header received
0928 21:53:14.780126 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.780127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.780139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.780197 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:14.780866 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.781369 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.782247 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.782425 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.782469 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.783247 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:14.783248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:14.783248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:14.783253 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:14.783269 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:14.783273 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:14.783319 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.783434 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.783464 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:53:14.783752 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.779264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:37259 (local address 127.3.0.195:45099)
0928 21:53:14.779463 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:14.779503 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:14.780043 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:14.780258 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:14.780262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:14.780270 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:14.780796 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.780801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.781432 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.781433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.782188 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.782190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.782487 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.782488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.782597 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.783106 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:53:14.783107 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:14.783137 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:53:14.783159 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:14.783352 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:14.783358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:14.783361 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:14.783433 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:53:14.783519 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260928 21:53:14.784178 22160 heartbeater.cc:347] Connected to a master server at 127.3.0.254:37259
I20260928 21:53:14.784256 22160 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:14.784425 22160 heartbeater.cc:511] Master 127.3.0.254:37259 requested a full tablet report, sending...
I20260928 21:53:14.784791 21710 ts_manager.cc:194] Registered new tserver with Master: f45adbfaed7248ad8d26d3e46ebb086b (127.3.0.195:46795)
I20260928 21:53:14.785127 21710 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.195:45099
I20260928 21:53:14.790737 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:53:14.794950 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.792065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.3.0.254:37259)
0928 21:53:14.792113 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.792115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.792237 (+ 122us) server_negotiation.cc:407] Connection header received
0928 21:53:14.792297 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.792298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.792308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.792343 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:14.792525 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.793065 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.793815 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.794005 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.794057 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.794663 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:14.794664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:14.794664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:14.794669 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:14.794683 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:14.794686 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:14.794761 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.794857 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.794868 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:53:14.796427 21710 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:53:14.797936 21710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:53:14.804025 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.800007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:38581 (local address 127.0.0.1:36660)
0928 21:53:14.800158 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:14.800174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:14.800235 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:14.800575 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:14.800579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:14.800581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:14.800647 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.800648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.801745 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.801746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.803008 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.803011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.803732 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.803734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.803808 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.803821 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 21:53:14.803895 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:53:14.803949 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":7,"thread_start_us":96,"threads_started":1}
I20260928 21:53:14.804033 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.800110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.3.0.193:38581)
0928 21:53:14.800389 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.800394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.800404 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:53:14.800443 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.800445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.800490 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.800564 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:14.800788 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.801664 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.803164 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.803656 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.803720 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.803828 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.803916 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.803957 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":84,"threads_started":1}
I20260928 21:53:14.804121 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.800201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:42927 (local address 127.0.0.1:39452)
0928 21:53:14.800351 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:14.800359 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:14.800394 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:14.801052 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:14.801053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:14.801055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:14.801089 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.801091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.802242 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.802243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.803005 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.803007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.803898 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.803900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.803970 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.803975 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:53:14.804005 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 21:53:14.804043 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":4,"thread_start_us":66,"threads_started":1}
I20260928 21:53:14.804426 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.800432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.3.0.194:42927)
0928 21:53:14.800906 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.800911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.800920 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:53:14.800955 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.800957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.800972 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.801046 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:14.801175 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.802172 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.803084 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.803803 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.804188 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.804285 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.804309 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.804348 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":96,"threads_started":1}
I20260928 21:53:14.804426 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.800292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.3.0.195:46795)
0928 21:53:14.800535 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:14.800539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:14.800547 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:53:14.800644 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:14.800647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:14.800662 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:14.800760 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:14.800855 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.801913 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.803102 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.803804 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.803869 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.803954 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:14.804317 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 21:53:14.804352 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":64,"threads_started":1}
I20260928 21:53:14.804262 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:14.800003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:46795 (local address 127.0.0.1:60092)
0928 21:53:14.800130 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:14.800172 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:14.800235 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:14.800759 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:14.800760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:14.800762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:14.800790 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.800791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.801991 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.801992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.803048 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:14.803050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:14.804081 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:14.804082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:14.804133 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:14.804136 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:14.804176 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 21:53:14.804208 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260928 21:53:14.804997 21963 tablet_service.cc:1467] Processing CreateTablet for tablet a06e32a5a78347149e623b9cf6bc3369 (DEFAULT_TABLE table=test-table [id=5bb1a9b9d71748feb7186b7176d53202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:14.805002 21832 tablet_service.cc:1467] Processing CreateTablet for tablet a06e32a5a78347149e623b9cf6bc3369 (DEFAULT_TABLE table=test-table [id=5bb1a9b9d71748feb7186b7176d53202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:14.805274 21832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06e32a5a78347149e623b9cf6bc3369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:14.805284 21963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06e32a5a78347149e623b9cf6bc3369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:14.807164 22094 tablet_service.cc:1467] Processing CreateTablet for tablet a06e32a5a78347149e623b9cf6bc3369 (DEFAULT_TABLE table=test-table [id=5bb1a9b9d71748feb7186b7176d53202]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:14.807475 22094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06e32a5a78347149e623b9cf6bc3369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:14.807919 22174 tablet_bootstrap.cc:492] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4: Bootstrap starting.
I20260928 21:53:14.808197 22175 tablet_bootstrap.cc:492] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: Bootstrap starting.
I20260928 21:53:14.808666 22174 tablet_bootstrap.cc:654] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:14.808808 22175 tablet_bootstrap.cc:654] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:14.808987 22174 log.cc:824] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:14.809103 22175 log.cc:824] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:14.809747 22174 tablet_bootstrap.cc:492] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4: No bootstrap required, opened a new log
I20260928 21:53:14.809813 22175 tablet_bootstrap.cc:492] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: No bootstrap required, opened a new log
I20260928 21:53:14.809818 22174 ts_tablet_manager.cc:1397] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:53:14.809876 22175 ts_tablet_manager.cc:1397] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:53:14.810413 22178 tablet_bootstrap.cc:492] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: Bootstrap starting.
I20260928 21:53:14.811044 22178 tablet_bootstrap.cc:654] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:14.811332 22178 log.cc:824] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:14.811825 22174 raft_consensus.cc:348] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 } } peers { permanent_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38581 } } peers { permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 } }
I20260928 21:53:14.811957 22174 raft_consensus.cc:374] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:14.811978 22174 raft_consensus.cc:729] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e25e9db70e4f1795ad3adae17f46f4, State: Initialized, Role: FOLLOWER
I20260928 21:53:14.812058 22178 tablet_bootstrap.cc:492] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: No bootstrap required, opened a new log
I20260928 21:53:14.812062 22174 consensus_queue.cc:260] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 } } peers { permanent_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38581 } } peers { permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 } }
I20260928 21:53:14.812114 22178 ts_tablet_manager.cc:1397] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:53:14.812077 22175 raft_consensus.cc:348] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 } } peers { permanent_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38581 } } peers { permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 } }
I20260928 21:53:14.812216 22175 raft_consensus.cc:374] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:14.812247 22175 raft_consensus.cc:729] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f549ff9e37214a2bbab08d0873d49276, State: Initialized, Role: FOLLOWER
I20260928 21:53:14.812287 22174 ts_tablet_manager.cc:1428] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:53:14.812346 22175 consensus_queue.cc:260] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 } } peers { permanent_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38581 } } peers { permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 } }
I20260928 21:53:14.812600 22175 ts_tablet_manager.cc:1428] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:53:14.812629 22029 heartbeater.cc:503] Master 127.3.0.254:37259 was elected leader, sending a full tablet report...
I20260928 21:53:14.812824 21898 heartbeater.cc:503] Master 127.3.0.254:37259 was elected leader, sending a full tablet report...
I20260928 21:53:14.814309 22178 raft_consensus.cc:348] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 } } peers { permanent_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38581 } } peers { permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 } }
I20260928 21:53:14.814422 22178 raft_consensus.cc:374] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:14.814451 22178 raft_consensus.cc:729] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45adbfaed7248ad8d26d3e46ebb086b, State: Initialized, Role: FOLLOWER
I20260928 21:53:14.814554 22178 consensus_queue.cc:260] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 } } peers { permanent_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38581 } } peers { permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 } }
I20260928 21:53:14.814774 22178 ts_tablet_manager.cc:1428] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:53:14.814841 22160 heartbeater.cc:503] Master 127.3.0.254:37259 was elected leader, sending a full tablet report...
W20260928 21:53:14.906977 22030 tablet.cc:2369] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:53:15.029079 22161 tablet.cc:2369] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:53:15.030339 21899 tablet.cc:2369] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:53:15.092808 22180 raft_consensus.cc:482] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:53:15.092962 22180 raft_consensus.cc:504] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 } } peers { permanent_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38581 } } peers { permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 } }
I20260928 21:53:15.093309 22180 leader_election.cc:288] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45adbfaed7248ad8d26d3e46ebb086b (127.3.0.195:46795), f549ff9e37214a2bbab08d0873d49276 (127.3.0.194:42927)
I20260928 21:53:15.095971 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:15.093554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:51671 (local address 127.3.0.195:46795)
0928 21:53:15.093614 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:15.093616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:15.093676 (+ 60us) server_negotiation.cc:407] Connection header received
0928 21:53:15.093744 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:15.093745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:15.093754 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:15.093801 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:15.093937 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:15.094400 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:15.095326 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:15.095669 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:15.095727 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:15.095739 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:15.095874 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:53:15.095912 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:53:15.095983 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:15.093478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:46795 (local address 127.3.0.193:51671)
0928 21:53:15.093593 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:15.093607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:15.093687 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:15.093806 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:15.093807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:15.093808 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:15.093879 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:15.093881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:15.094472 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:15.094473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:15.095238 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:15.095240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:15.095741 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:15.095742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:15.095807 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:15.095810 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:15.095853 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:53:15.095891 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260928 21:53:15.096139 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:15.093718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:47985 (local address 127.3.0.194:42927)
0928 21:53:15.093763 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:15.093765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:15.093826 (+ 61us) server_negotiation.cc:407] Connection header received
0928 21:53:15.093903 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:15.093903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:15.093911 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:15.093953 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:15.094097 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:15.094549 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:15.095453 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:15.095817 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:15.095875 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:15.095884 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:15.096040 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:53:15.096077 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:53:15.096149 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:15.093642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:42927 (local address 127.3.0.193:47985)
0928 21:53:15.093771 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:15.093780 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:15.093841 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:15.093963 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:15.093964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:15.093965 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:15.094037 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:15.094039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:15.094618 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:15.094620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:15.095360 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:15.095362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:15.095907 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:15.095908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:15.095981 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:15.095984 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:15.096028 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:53:15.096066 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":1,"thread_start_us":41,"threads_started":1}
I20260928 21:53:15.096534 21983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06e32a5a78347149e623b9cf6bc3369" candidate_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f549ff9e37214a2bbab08d0873d49276" is_pre_election: true
I20260928 21:53:15.096539 22114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06e32a5a78347149e623b9cf6bc3369" candidate_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" is_pre_election: true
I20260928 21:53:15.096668 21983 raft_consensus.cc:2454] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e25e9db70e4f1795ad3adae17f46f4 in term 0.
I20260928 21:53:15.096673 22114 raft_consensus.cc:2454] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e25e9db70e4f1795ad3adae17f46f4 in term 0.
I20260928 21:53:15.096870 21788 leader_election.cc:302] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0e25e9db70e4f1795ad3adae17f46f4, f45adbfaed7248ad8d26d3e46ebb086b; no voters:
I20260928 21:53:15.096984 22180 raft_consensus.cc:2781] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:53:15.097039 22180 raft_consensus.cc:482] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:53:15.097067 22180 raft_consensus.cc:3037] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:15.097728 22180 raft_consensus.cc:504] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 } } peers { permanent_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38581 } } peers { permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 } }
I20260928 21:53:15.097831 22180 leader_election.cc:288] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [CANDIDATE]: Term 1 election: Requested vote from peers f45adbfaed7248ad8d26d3e46ebb086b (127.3.0.195:46795), f549ff9e37214a2bbab08d0873d49276 (127.3.0.194:42927)
I20260928 21:53:15.098027 21983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06e32a5a78347149e623b9cf6bc3369" candidate_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f549ff9e37214a2bbab08d0873d49276"
I20260928 21:53:15.098050 22114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06e32a5a78347149e623b9cf6bc3369" candidate_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45adbfaed7248ad8d26d3e46ebb086b"
I20260928 21:53:15.098091 21983 raft_consensus.cc:3037] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:15.098119 22114 raft_consensus.cc:3037] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:15.098752 21983 raft_consensus.cc:2454] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e25e9db70e4f1795ad3adae17f46f4 in term 1.
I20260928 21:53:15.098877 22114 raft_consensus.cc:2454] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e25e9db70e4f1795ad3adae17f46f4 in term 1.
I20260928 21:53:15.099017 21788 leader_election.cc:302] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0e25e9db70e4f1795ad3adae17f46f4, f549ff9e37214a2bbab08d0873d49276; no voters:
I20260928 21:53:15.099093 22180 raft_consensus.cc:2781] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:15.099251 22180 raft_consensus.cc:686] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [term 1 LEADER]: Becoming Leader. State: Replica: b0e25e9db70e4f1795ad3adae17f46f4, State: Running, Role: LEADER
I20260928 21:53:15.099346 22180 consensus_queue.cc:237] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 } } peers { permanent_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38581 } } peers { permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 } }
I20260928 21:53:15.100134 21710 catalog_manager.cc:5495] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0e25e9db70e4f1795ad3adae17f46f4 (127.3.0.193). New cstate: current_term: 1 leader_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0e25e9db70e4f1795ad3adae17f46f4" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 38581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 } health_report { overall_health: UNKNOWN } } }
I20260928 21:53:15.156620 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:15.154392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.3.0.193:38581)
0928 21:53:15.154449 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:15.154451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:15.154460 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:53:15.154535 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:15.154536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:15.154557 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:15.154617 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:15.154738 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:15.155238 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:15.155958 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:15.156142 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:15.156193 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:15.156300 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:15.156387 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:15.156482 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:15.156527 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 21:53:15.156540 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:53:15.158517 21983 raft_consensus.cc:1261] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276 [term 1 FOLLOWER]: Refusing update from remote peer b0e25e9db70e4f1795ad3adae17f46f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:15.158516 22114 raft_consensus.cc:1261] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b [term 1 FOLLOWER]: Refusing update from remote peer b0e25e9db70e4f1795ad3adae17f46f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:15.158900 22185 consensus_queue.cc:1035] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45adbfaed7248ad8d26d3e46ebb086b" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 46795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:15.159041 22180 consensus_queue.cc:1035] T a06e32a5a78347149e623b9cf6bc3369 P b0e25e9db70e4f1795ad3adae17f46f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f549ff9e37214a2bbab08d0873d49276" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:15.162176 22190 mvcc.cc:205] Tried to move back new op lower bound from 7334430290566643712 to 7334430290327584768. Current Snapshot: MvccSnapshot[applied={T|T < 7334430290566643712}]
I20260928 21:53:15.163645 21709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:15.163735 21709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:15.164435 21709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fa64d37c602471a8b08011370a64273: Sending DeleteTablet for 3 replicas of tablet a06e32a5a78347149e623b9cf6bc3369
I20260928 21:53:15.164785 21963 tablet_service.cc:1514] Processing DeleteTablet for tablet a06e32a5a78347149e623b9cf6bc3369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:15 UTC) from {username='slave'} at 127.0.0.1:39452
I20260928 21:53:15.164759 22094 tablet_service.cc:1514] Processing DeleteTablet for tablet a06e32a5a78347149e623b9cf6bc3369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:15 UTC) from {username='slave'} at 127.0.0.1:60092
I20260928 21:53:15.164824 21832 tablet_service.cc:1514] Processing DeleteTablet for tablet a06e32a5a78347149e623b9cf6bc3369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:15 UTC) from {username='slave'} at 127.0.0.1:36660
I20260928 21:53:15.164963 22198 tablet_replica.cc:330] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: stopping tablet replica
I20260928 21:53:15.165158 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 21770
I20260928 21:53:15.165172 22198 raft_consensus.cc:2229] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:15.165272 22198 raft_consensus.cc:2258] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:15.166353 22199 tablet_replica.cc:330] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: stopping tablet replica
I20260928 21:53:15.166446 22199 raft_consensus.cc:2229] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:15.166548 22199 raft_consensus.cc:2258] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:15.166909 22199 ts_tablet_manager.cc:1905] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:15.167007 22198 ts_tablet_manager.cc:1905] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:15.168282 22199 ts_tablet_manager.cc:1918] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:15.168344 22199 log.cc:1196] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/a06e32a5a78347149e623b9cf6bc3369
I20260928 21:53:15.168553 22198 ts_tablet_manager.cc:1918] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:15.168591 22199 ts_tablet_manager.cc:1939] T a06e32a5a78347149e623b9cf6bc3369 P f45adbfaed7248ad8d26d3e46ebb086b: Deleting consensus metadata
I20260928 21:53:15.168622 22198 log.cc:1196] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/a06e32a5a78347149e623b9cf6bc3369
I20260928 21:53:15.169052 22198 ts_tablet_manager.cc:1939] T a06e32a5a78347149e623b9cf6bc3369 P f549ff9e37214a2bbab08d0873d49276: Deleting consensus metadata
I20260928 21:53:15.169068 21697 catalog_manager.cc:4841] TS f45adbfaed7248ad8d26d3e46ebb086b (127.3.0.195:46795): tablet a06e32a5a78347149e623b9cf6bc3369 (table test-table [id=5bb1a9b9d71748feb7186b7176d53202]) successfully deleted
I20260928 21:53:15.169548 21697 catalog_manager.cc:4841] TS f549ff9e37214a2bbab08d0873d49276 (127.3.0.194:42927): tablet a06e32a5a78347149e623b9cf6bc3369 (table test-table [id=5bb1a9b9d71748feb7186b7176d53202]) successfully deleted
W20260928 21:53:15.171383 21694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:38581 (error 108)
I20260928 21:53:15.171622 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 21901
I20260928 21:53:15.172188 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:15.171754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:38581 (local address 127.0.0.1:36692)
0928 21:53:15.172115 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:15.172155 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
W20260928 21:53:15.173709 21694 catalog_manager.cc:4584] TS b0e25e9db70e4f1795ad3adae17f46f4 (127.3.0.193:38581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a06e32a5a78347149e623b9cf6bc3369: Network error: Client connection negotiation failed: client connection to 127.3.0.193:38581: connect: Connection refused (error 111)
I20260928 21:53:15.175714 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 22032
I20260928 21:53:15.179796 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 21679
2026-09-28T21:53:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (927 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:15 dist-test-slave-jrck krb5kdc[22204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:15 dist-test-slave-jrck krb5kdc[22204](info): set up 2 sockets
Sep 28 21:53:15 dist-test-slave-jrck krb5kdc[22204](info): commencing operation
krb5kdc: starting...
W20260928 21:53:16.124001 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:16 dist-test-slave-jrck krb5kdc[22204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:16.233037 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:40397
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:16.310791 22217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:16.310997 22217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:16.311018 22217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:16.312141 22217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:16.312181 22217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:16.312194 22217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:16.312206 22217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:16.312215 22217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:16.313689 22217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:40397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:16.313920 22217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:16.314128 22217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:16.318639 22225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:16.318677 22223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:16.318639 22222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:16.318836 22217 server_base.cc:1034] running on GCE node
I20260928 21:53:16.319097 22217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:16.319373 22217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:16.320505 22217 hybrid_clock.cc:648] HybridClock initialized: now 1790632396320489 us; error 25 us; skew 500 ppm
Sep 28 21:53:16 dist-test-slave-jrck krb5kdc[22204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:16.322865 22217 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:16.323342 22217 webserver.cc:460] Webserver started at http://127.3.0.254:41355/ using document root <none> and password file <none>
I20260928 21:53:16.323539 22217 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:16.323586 22217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:16.323706 22217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:16.324792 22217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "9b768994b2f84b2ab09ab1a231e10459"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "ec36b24c6b7d5f1222c1b89892b91b5d"
server_key_iv: "181d3da26600981c798279016b44f5c2"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.325121 22217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "9b768994b2f84b2ab09ab1a231e10459"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "ec36b24c6b7d5f1222c1b89892b91b5d"
server_key_iv: "181d3da26600981c798279016b44f5c2"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.326333 22217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:53:16.326990 22232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:16.327138 22217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:16.327217 22217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "9b768994b2f84b2ab09ab1a231e10459"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "ec36b24c6b7d5f1222c1b89892b91b5d"
server_key_iv: "181d3da26600981c798279016b44f5c2"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.327275 22217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:16.338930 22217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:16.341243 22217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:16.341377 22217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:16.345103 22217 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:40397
I20260928 21:53:16.345136 22284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:40397 every 8 connection(s)
I20260928 21:53:16.345459 22217 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:16.346289 22285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:16.347299 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 22217
I20260928 21:53:16.347420 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:16.347617 3075 external_mini_cluster.cc:1428] Setting key c61c98664157753808eb92b2b8933177
I20260928 21:53:16.349035 22285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459: Bootstrap starting.
I20260928 21:53:16.349747 22285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:16.350023 22285 log.cc:824] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:16.350685 22285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459: No bootstrap required, opened a new log
I20260928 21:53:16.352960 22285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b768994b2f84b2ab09ab1a231e10459" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40397 } }
I20260928 21:53:16.353135 22285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:16.353171 22285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b768994b2f84b2ab09ab1a231e10459, State: Initialized, Role: FOLLOWER
I20260928 21:53:16.353308 22285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b768994b2f84b2ab09ab1a231e10459" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40397 } }
I20260928 21:53:16.353389 22285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:16.353427 22285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:16.353477 22285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:16.354082 22285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b768994b2f84b2ab09ab1a231e10459" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40397 } }
I20260928 21:53:16.354171 22285 leader_election.cc:302] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b768994b2f84b2ab09ab1a231e10459; no voters:
Sep 28 21:53:16 dist-test-slave-jrck krb5kdc[22204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:16.354315 22285 leader_election.cc:288] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:16.354355 22290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:16.354507 22290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [term 1 LEADER]: Becoming Leader. State: Replica: 9b768994b2f84b2ab09ab1a231e10459, State: Running, Role: LEADER
I20260928 21:53:16.354570 22285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:16.354677 22290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b768994b2f84b2ab09ab1a231e10459" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40397 } }
I20260928 21:53:16.355011 22290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b768994b2f84b2ab09ab1a231e10459. Latest consensus state: current_term: 1 leader_uuid: "9b768994b2f84b2ab09ab1a231e10459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b768994b2f84b2ab09ab1a231e10459" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40397 } } }
I20260928 21:53:16.355080 22290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:16.355240 22292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b768994b2f84b2ab09ab1a231e10459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b768994b2f84b2ab09ab1a231e10459" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40397 } } }
I20260928 21:53:16.355322 22292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:16.356047 22299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:16.356676 22299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:16.358297 22299 catalog_manager.cc:1347] Generated new cluster ID: 3b21b2e30d2e4664aba3eae4f495e005
I20260928 21:53:16.358345 22299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:16.358474 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.348125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.3.0.254:40397)
0928 21:53:16.348279 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:16.348289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:16.348317 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:53:16.348807 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:16.348814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:16.348862 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:16.348983 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:16.349715 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.350768 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.351793 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.352192 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.352263 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.354905 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:16.354910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:16.354910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:16.354923 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:16.356955 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:16.357277 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:16.357280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:16.357280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:16.357388 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:16.357620 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:16.357620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:16.357621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:16.357917 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:16.358059 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:16.358174 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 21:53:16.358318 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260928 21:53:16.366104 22299 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:16.366816 22299 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:16.372952 22299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459: Generated new TSK 0
I20260928 21:53:16.373143 22299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.0.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
Entry for principal kudu/127.3.0.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab.
I20260928 21:53:16.399533 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40397
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:16.481217 22311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:16.481427 22311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:16.481449 22311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:16.482656 22311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:16.482702 22311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:16.482729 22311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:53:16.484253 22311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.193.keytab
--principal=kudu/127.3.0.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:16.484496 22311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:16.484714 22311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:16.489485 22316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:16.489569 22311 server_base.cc:1034] running on GCE node
W20260928 21:53:16.489485 22317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:16.489594 22319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:16.489880 22311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:16.490123 22311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:16.491273 22311 hybrid_clock.cc:648] HybridClock initialized: now 1790632396491253 us; error 33 us; skew 500 ppm
Sep 28 21:53:16 dist-test-slave-jrck krb5kdc[22204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:16.493739 22311 init.cc:377] Logged in from keytab as kudu/127.3.0.193@KRBTEST.COM (short username kudu)
I20260928 21:53:16.494234 22311 webserver.cc:460] Webserver started at http://127.3.0.193:33915/ using document root <none> and password file <none>
I20260928 21:53:16.494455 22311 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:16.494504 22311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:16.494623 22311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:16.495738 22311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "d61032784d2d4d69a0b8f062d0b47d23"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "4445f75be2add97b2912d8f5fed5734b"
server_key_iv: "54ca9bc12017d717fc3664679a2a1378"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.496078 22311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "d61032784d2d4d69a0b8f062d0b47d23"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "4445f75be2add97b2912d8f5fed5734b"
server_key_iv: "54ca9bc12017d717fc3664679a2a1378"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.497364 22311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:16.498162 22326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:16.498360 22311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:53:16.498442 22311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "d61032784d2d4d69a0b8f062d0b47d23"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "4445f75be2add97b2912d8f5fed5734b"
server_key_iv: "54ca9bc12017d717fc3664679a2a1378"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.498508 22311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:16.515409 22311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:16.517450 22311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:16.517598 22311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:16.517841 22311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:16.518136 22311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:16.518168 22311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:16.518198 22311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:16.518215 22311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:16.523841 22311 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:44201
I20260928 21:53:16.523891 22439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:44201 every 8 connection(s)
I20260928 21:53:16.524210 22311 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:53:16.524536 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 22311
I20260928 21:53:16.524643 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:53:16.524854 3075 external_mini_cluster.cc:1428] Setting key 6e6fdd71c887f3510338f2dfd4ff5961
Sep 28 21:53:16 dist-test-slave-jrck krb5kdc[22204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.193@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.0.194@KRBTEST.COM; defaulting to no policy
I20260928 21:53:16.534255 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.524940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:51547 (local address 127.3.0.254:40397)
0928 21:53:16.524990 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:16.524992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:16.525215 (+ 223us) server_negotiation.cc:407] Connection header received
0928 21:53:16.525757 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:16.525758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:16.525770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:16.525808 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:16.526762 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.527283 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.528687 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.528964 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.529021 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.531559 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:16.531560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:16.531561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:16.531567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:16.533083 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:16.533446 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:16.533447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:16.533447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:16.533493 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:16.533698 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:16.533699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:16.533699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:16.533840 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:16.533892 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:16.534116 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:53:16.534187 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:53:16.534442 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.524922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:40397 (local address 127.3.0.193:51547)
0928 21:53:16.525090 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:16.525141 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:16.525668 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:16.525892 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:16.525897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:16.526454 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:16.526654 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.526659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.527372 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.527373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.528587 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.528590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.529049 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.529050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.529194 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.529742 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:16.529753 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:16.531450 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:16.533156 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:16.533162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:16.533165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:16.533386 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:16.533543 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:16.533543 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:16.533544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:16.533652 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:16.533891 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:16.533894 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:16.534015 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 21:53:16.534096 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:53:16.534231 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.194@KRBTEST.COM" created.
I20260928 21:53:16.534933 22440 heartbeater.cc:347] Connected to a master server at 127.3.0.254:40397
I20260928 21:53:16.535037 22440 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:16.535225 22440 heartbeater.cc:511] Master 127.3.0.254:40397 requested a full tablet report, sending...
I20260928 21:53:16.535773 22249 ts_manager.cc:194] Registered new tserver with Master: d61032784d2d4d69a0b8f062d0b47d23 (127.3.0.193:44201)
I20260928 21:53:16.536352 22249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.193@KRBTEST.COM'} at 127.3.0.193:51547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
Entry for principal kudu/127.3.0.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab.
I20260928 21:53:16.548830 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40397
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:16.629618 22445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:16.629853 22445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:16.629887 22445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:16.631095 22445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:16.631143 22445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:16.631179 22445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:53:16.632668 22445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.194.keytab
--principal=kudu/127.3.0.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:16.632899 22445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:16.633108 22445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:16.641273 22453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:16.641636 22451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:16.641847 22450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:16.642343 22445 server_base.cc:1034] running on GCE node
I20260928 21:53:16.642589 22445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:16.642865 22445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:16.644016 22445 hybrid_clock.cc:648] HybridClock initialized: now 1790632396644005 us; error 37 us; skew 500 ppm
Sep 28 21:53:16 dist-test-slave-jrck krb5kdc[22204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:16.646410 22445 init.cc:377] Logged in from keytab as kudu/127.3.0.194@KRBTEST.COM (short username kudu)
I20260928 21:53:16.646822 22445 webserver.cc:460] Webserver started at http://127.3.0.194:43443/ using document root <none> and password file <none>
I20260928 21:53:16.647048 22445 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:16.647096 22445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:16.647219 22445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:16.648247 22445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "bdee04250b2c42cfaa979d39cd851aef"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "8eb1670626c9bd5a0b3a3e8b908a4841"
server_key_iv: "99e6a3f25b90aede0b20f643a114f2f0"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.648593 22445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "bdee04250b2c42cfaa979d39cd851aef"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "8eb1670626c9bd5a0b3a3e8b908a4841"
server_key_iv: "99e6a3f25b90aede0b20f643a114f2f0"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.649781 22445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:16.650414 22460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:16.650597 22445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:16.650691 22445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "bdee04250b2c42cfaa979d39cd851aef"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "8eb1670626c9bd5a0b3a3e8b908a4841"
server_key_iv: "99e6a3f25b90aede0b20f643a114f2f0"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.650755 22445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:16.663872 22445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:16.666116 22445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:16.666257 22445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:16.666471 22445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:16.666772 22445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:16.666805 22445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:16.666842 22445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:16.666867 22445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:16.672255 22445 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:42841
I20260928 21:53:16.672319 22573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:42841 every 8 connection(s)
I20260928 21:53:16.672626 22445 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:53:16.673513 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 22445
I20260928 21:53:16.673626 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:53:16.673806 3075 external_mini_cluster.cc:1428] Setting key a49b4d2c0ce39770211014a1baa0626b
Sep 28 21:53:16 dist-test-slave-jrck krb5kdc[22204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.194@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.0.195@KRBTEST.COM; defaulting to no policy
I20260928 21:53:16.682853 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.673300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:38551 (local address 127.3.0.254:40397)
0928 21:53:16.673355 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:16.673357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:16.673544 (+ 187us) server_negotiation.cc:407] Connection header received
0928 21:53:16.674118 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:16.674119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:16.674130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:16.674188 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:16.675244 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.675930 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.677429 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.677627 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.677690 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.680140 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:16.680142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:16.680142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:16.680149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:16.681496 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:16.681973 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:16.681974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:16.681974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:16.682030 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:16.682252 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:16.682252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:16.682253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:16.682351 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:16.682395 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:16.682691 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 21:53:16.682780 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:53:16.683038 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.673275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:40397 (local address 127.3.0.194:38551)
0928 21:53:16.673438 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:16.673474 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:16.674012 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:16.674273 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:16.674277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:16.674944 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:16.675144 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.675149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.676033 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.676034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.677326 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.677329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.677710 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.677710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.677867 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.678367 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:16.678379 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:16.680030 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:16.681582 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:16.681589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:16.681592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:16.681899 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:16.682089 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:16.682090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:16.682090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:16.682200 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:16.682422 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:16.682428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:16.682560 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 21:53:16.682668 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:53:16.682809 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":119,"threads_started":1}
I20260928 21:53:16.683526 22574 heartbeater.cc:347] Connected to a master server at 127.3.0.254:40397
I20260928 21:53:16.683648 22574 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:16.683873 22574 heartbeater.cc:511] Master 127.3.0.254:40397 requested a full tablet report, sending...
I20260928 21:53:16.684281 22249 ts_manager.cc:194] Registered new tserver with Master: bdee04250b2c42cfaa979d39cd851aef (127.3.0.194:42841)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.195@KRBTEST.COM" created.
I20260928 21:53:16.684770 22249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.194@KRBTEST.COM'} at 127.3.0.194:38551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
Entry for principal kudu/127.3.0.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab.
I20260928 21:53:16.698975 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40397
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:16.782160 22579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:16.782393 22579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:16.782418 22579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:16.783617 22579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:16.783663 22579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:16.783689 22579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:53:16.785192 22579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.195.keytab
--principal=kudu/127.3.0.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:16.785393 22579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:16.785576 22579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:16.790683 22587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:16.790704 22585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:16.790693 22584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:16.790755 22579 server_base.cc:1034] running on GCE node
I20260928 21:53:16.791066 22579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:16.791324 22579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:16.792531 22579 hybrid_clock.cc:648] HybridClock initialized: now 1790632396792505 us; error 34 us; skew 500 ppm
Sep 28 21:53:16 dist-test-slave-jrck krb5kdc[22204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:16.795123 22579 init.cc:377] Logged in from keytab as kudu/127.3.0.195@KRBTEST.COM (short username kudu)
I20260928 21:53:16.795573 22579 webserver.cc:460] Webserver started at http://127.3.0.195:40795/ using document root <none> and password file <none>
I20260928 21:53:16.795783 22579 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:16.795841 22579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:16.795981 22579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:16.797165 22579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "580242de72764f9caaef4e698b7c1c3e"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "1634cba893e5bafc2b9809e22bf6aee1"
server_key_iv: "327b0e829450481f0d9d09047a84d413"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.797502 22579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "580242de72764f9caaef4e698b7c1c3e"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "1634cba893e5bafc2b9809e22bf6aee1"
server_key_iv: "327b0e829450481f0d9d09047a84d413"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.798892 22579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:53:16.799666 22594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:16.799851 22579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:16.799938 22579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "580242de72764f9caaef4e698b7c1c3e"
format_stamp: "Formatted at 2026-09-28 21:53:16 on dist-test-slave-jrck"
server_key: "1634cba893e5bafc2b9809e22bf6aee1"
server_key_iv: "327b0e829450481f0d9d09047a84d413"
server_key_version: "encryptionkey@0"
I20260928 21:53:16.800001 22579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:16.808130 22579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:16.810245 22579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:16.810384 22579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:16.810604 22579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:16.810902 22579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:16.810935 22579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:16.810963 22579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:16.810981 22579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:16.816264 22579 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:41189
I20260928 21:53:16.816327 22707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:41189 every 8 connection(s)
I20260928 21:53:16.816628 22579 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:53:16 dist-test-slave-jrck krb5kdc[22204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632396, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.195@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:16.823980 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 22579
I20260928 21:53:16.824079 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:53:16.824249 3075 external_mini_cluster.cc:1428] Setting key 3c1ee182b9cf90d601b223c801dc84cb
I20260928 21:53:16.825294 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.817354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:47543 (local address 127.3.0.254:40397)
0928 21:53:16.817432 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:16.817434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:16.817574 (+ 140us) server_negotiation.cc:407] Connection header received
0928 21:53:16.818084 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:16.818085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:16.818094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:16.818143 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:16.819041 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.819540 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.820589 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.820789 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.820846 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.822814 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:16.822815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:16.822816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:16.822824 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:16.823861 (+ 1037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:16.824282 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:16.824283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:16.824284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:16.824347 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:16.824661 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:16.824662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:16.824662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:16.824804 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:16.824870 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:16.825169 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 21:53:16.825232 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:53:16.825459 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.817314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:40397 (local address 127.3.0.195:47543)
0928 21:53:16.817460 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:16.817497 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:16.818004 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:16.818199 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:16.818202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:16.818766 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:16.818963 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.818967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.819624 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.819625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.820492 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.820495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.820860 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.820861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.820984 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.821423 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:16.821430 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:16.822723 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:16.823945 (+ 1222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:16.823950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:16.823952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:16.824219 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:16.824434 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:16.824435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:16.824436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:16.824597 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:16.824865 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:16.824872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:16.825035 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 21:53:16.825142 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 21:53:16.825270 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260928 21:53:16.825841 22708 heartbeater.cc:347] Connected to a master server at 127.3.0.254:40397
I20260928 21:53:16.825925 22708 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:16.826088 22708 heartbeater.cc:511] Master 127.3.0.254:40397 requested a full tablet report, sending...
I20260928 21:53:16.826426 22249 ts_manager.cc:194] Registered new tserver with Master: 580242de72764f9caaef4e698b7c1c3e (127.3.0.195:41189)
I20260928 21:53:16.826781 22249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.0.195@KRBTEST.COM'} at 127.3.0.195:47543
I20260928 21:53:16.836122 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:53:16.843291 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.837475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.3.0.254:40397)
0928 21:53:16.837534 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:16.837536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:16.837613 (+ 77us) server_negotiation.cc:407] Connection header received
0928 21:53:16.837682 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:16.837683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:16.837692 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:16.837736 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:16.838273 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.838760 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.839516 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.839710 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.839763 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.840927 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:16.840928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:16.840933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:16.840940 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:16.842256 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:16.842558 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:16.842559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:16.842560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:16.842610 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:16.842774 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:16.842774 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:16.842775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:16.842919 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:16.843006 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:16.843125 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 21:53:16.843201 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:53:16.845053 22249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:53:16.846591 22249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:53:16.852944 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.848793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:42841 (local address 127.0.0.1:42186)
0928 21:53:16.848957 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:16.848999 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:16.849072 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:16.849383 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:16.849386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:16.849389 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:16.849463 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.849466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.850617 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.850618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.851798 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.851801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.852637 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.852638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.852727 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.852742 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:53:16.852812 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:53:16.852868 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":115,"threads_started":1}
I20260928 21:53:16.852954 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.848919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.3.0.194:42841)
0928 21:53:16.849076 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:16.849081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:16.849092 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:53:16.849163 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:16.849165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:16.849181 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:16.849245 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:16.849715 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.850508 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.851882 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.852568 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.852631 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.852693 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:16.852833 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 21:53:16.852871 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260928 21:53:16.853228 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.848797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:41189 (local address 127.0.0.1:51104)
0928 21:53:16.849259 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:16.849269 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:16.849332 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:16.849717 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:16.849718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:16.849720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:16.849770 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.849772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.851174 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.851176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.852163 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.852165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.853010 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.853012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.853081 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.853086 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:53:16.853116 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 21:53:16.853157 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"spinlock_wait_cycles":896,"thread_start_us":114,"threads_started":1}
I20260928 21:53:16.853611 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.849098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.3.0.195:41189)
0928 21:53:16.849252 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:16.849258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:16.849568 (+ 310us) server_negotiation.cc:407] Connection header received
0928 21:53:16.849614 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:16.849616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:16.849631 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:16.849711 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:16.849833 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.851007 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.852243 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.852923 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.853388 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.853461 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:16.853485 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 21:53:16.853528 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260928 21:53:16.853658 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.849050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:44201 (local address 127.0.0.1:53266)
0928 21:53:16.849421 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:16.849437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:16.849477 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:16.849624 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:16.849624 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:16.849626 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:16.849672 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.849674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.851044 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.851045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.852066 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:16.852068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.853456 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.853457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:16.853520 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.853523 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:16.853563 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 21:53:16.853599 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":62,"threads_started":1}
I20260928 21:53:16.853715 22508 tablet_service.cc:1467] Processing CreateTablet for tablet 0123b161ca3643bface36d34282ab698 (DEFAULT_TABLE table=test-table [id=9dadaaf40c4e4696be8f24fe0cabe044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:16.854003 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:16.849257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.3.0.193:44201)
0928 21:53:16.849451 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:16.849456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:16.849467 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:53:16.849540 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:16.849542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:16.849556 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:16.849628 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:16.849759 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.850917 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.852140 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:16.852884 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:16.852952 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:16.853837 (+ 885us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:16.853863 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 21:53:16.853913 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260928 21:53:16.854141 22508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0123b161ca3643bface36d34282ab698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:16.854281 22642 tablet_service.cc:1467] Processing CreateTablet for tablet 0123b161ca3643bface36d34282ab698 (DEFAULT_TABLE table=test-table [id=9dadaaf40c4e4696be8f24fe0cabe044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:16.854625 22642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0123b161ca3643bface36d34282ab698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:16.854625 22374 tablet_service.cc:1467] Processing CreateTablet for tablet 0123b161ca3643bface36d34282ab698 (DEFAULT_TABLE table=test-table [id=9dadaaf40c4e4696be8f24fe0cabe044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:16.854985 22374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0123b161ca3643bface36d34282ab698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:16.857479 22722 tablet_bootstrap.cc:492] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: Bootstrap starting.
I20260928 21:53:16.857904 22723 tablet_bootstrap.cc:492] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: Bootstrap starting.
I20260928 21:53:16.858107 22722 tablet_bootstrap.cc:654] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:16.858261 22724 tablet_bootstrap.cc:492] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23: Bootstrap starting.
I20260928 21:53:16.858672 22722 log.cc:824] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:16.858780 22724 tablet_bootstrap.cc:654] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:16.858804 22723 tablet_bootstrap.cc:654] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:16.859217 22724 log.cc:824] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:16.859236 22723 log.cc:824] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:16.859829 22724 tablet_bootstrap.cc:492] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23: No bootstrap required, opened a new log
I20260928 21:53:16.859869 22722 tablet_bootstrap.cc:492] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: No bootstrap required, opened a new log
I20260928 21:53:16.859889 22724 ts_tablet_manager.cc:1397] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:53:16.859923 22722 ts_tablet_manager.cc:1397] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:53:16.860836 22723 tablet_bootstrap.cc:492] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: No bootstrap required, opened a new log
I20260928 21:53:16.860903 22723 ts_tablet_manager.cc:1397] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:53:16.861330 22724 raft_consensus.cc:348] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 } } peers { permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 } } peers { permanent_uuid: "d61032784d2d4d69a0b8f062d0b47d23" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44201 } }
I20260928 21:53:16.861472 22724 raft_consensus.cc:374] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:16.861502 22724 raft_consensus.cc:729] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61032784d2d4d69a0b8f062d0b47d23, State: Initialized, Role: FOLLOWER
I20260928 21:53:16.861641 22724 consensus_queue.cc:260] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 } } peers { permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 } } peers { permanent_uuid: "d61032784d2d4d69a0b8f062d0b47d23" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44201 } }
I20260928 21:53:16.861728 22722 raft_consensus.cc:348] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 } } peers { permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 } } peers { permanent_uuid: "d61032784d2d4d69a0b8f062d0b47d23" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44201 } }
I20260928 21:53:16.861865 22724 ts_tablet_manager.cc:1428] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:53:16.861896 22722 raft_consensus.cc:374] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:16.861935 22722 raft_consensus.cc:729] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdee04250b2c42cfaa979d39cd851aef, State: Initialized, Role: FOLLOWER
I20260928 21:53:16.861965 22440 heartbeater.cc:503] Master 127.3.0.254:40397 was elected leader, sending a full tablet report...
I20260928 21:53:16.862053 22722 consensus_queue.cc:260] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 } } peers { permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 } } peers { permanent_uuid: "d61032784d2d4d69a0b8f062d0b47d23" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44201 } }
I20260928 21:53:16.862308 22722 ts_tablet_manager.cc:1428] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 21:53:16.862404 22574 heartbeater.cc:503] Master 127.3.0.254:40397 was elected leader, sending a full tablet report...
I20260928 21:53:16.862670 22723 raft_consensus.cc:348] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 } } peers { permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 } } peers { permanent_uuid: "d61032784d2d4d69a0b8f062d0b47d23" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44201 } }
I20260928 21:53:16.862811 22723 raft_consensus.cc:374] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:16.862831 22723 raft_consensus.cc:729] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580242de72764f9caaef4e698b7c1c3e, State: Initialized, Role: FOLLOWER
I20260928 21:53:16.862931 22723 consensus_queue.cc:260] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 } } peers { permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 } } peers { permanent_uuid: "d61032784d2d4d69a0b8f062d0b47d23" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44201 } }
I20260928 21:53:16.863178 22708 heartbeater.cc:503] Master 127.3.0.254:40397 was elected leader, sending a full tablet report...
I20260928 21:53:16.863171 22723 ts_tablet_manager.cc:1428] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260928 21:53:16.923168 22575 tablet.cc:2369] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:53:17.025000 22441 tablet.cc:2369] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:53:17.067180 22709 tablet.cc:2369] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:53:17.233806 22728 raft_consensus.cc:482] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:53:17.233955 22728 raft_consensus.cc:504] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 } } peers { permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 } } peers { permanent_uuid: "d61032784d2d4d69a0b8f062d0b47d23" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44201 } }
I20260928 21:53:17.234318 22728 leader_election.cc:288] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdee04250b2c42cfaa979d39cd851aef (127.3.0.194:42841), 580242de72764f9caaef4e698b7c1c3e (127.3.0.195:41189)
I20260928 21:53:17.238332 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.234581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:35121 (local address 127.3.0.195:41189)
0928 21:53:17.234629 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.234631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.234687 (+ 56us) server_negotiation.cc:407] Connection header received
0928 21:53:17.234800 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.234801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.234811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.234884 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:17.235072 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.235850 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.237424 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.238001 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.238058 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.238066 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.238218 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.238259 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:53:17.238332 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.234610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:51707 (local address 127.3.0.194:42841)
0928 21:53:17.234647 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.234649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.234700 (+ 51us) server_negotiation.cc:407] Connection header received
0928 21:53:17.234797 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.234798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.234811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.234884 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:17.235072 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.235819 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.237419 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.237972 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.238048 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.238059 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.238218 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.238268 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:53:17.238385 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.234503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:42841 (local address 127.3.0.193:51707)
0928 21:53:17.234626 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.234636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:17.234706 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:17.234905 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:17.234906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:17.234908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:17.235011 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.235012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.235973 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.235974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.237331 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.237335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.238060 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.238060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.238146 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.238152 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:53:17.238208 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:53:17.238263 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":80,"thread_start_us":59,"threads_started":1}
I20260928 21:53:17.238399 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.234503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:41189 (local address 127.3.0.193:35121)
0928 21:53:17.234627 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.234636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:17.234702 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:17.234918 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:17.234919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:17.234920 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:17.235006 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.235008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.235973 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.235974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.237332 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.237335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.238066 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.238067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.238146 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.238154 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:53:17.238208 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:53:17.238263 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":52,"thread_start_us":59,"threads_started":1}
I20260928 21:53:17.238762 22662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0123b161ca3643bface36d34282ab698" candidate_uuid: "d61032784d2d4d69a0b8f062d0b47d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580242de72764f9caaef4e698b7c1c3e" is_pre_election: true
I20260928 21:53:17.238839 22528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0123b161ca3643bface36d34282ab698" candidate_uuid: "d61032784d2d4d69a0b8f062d0b47d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdee04250b2c42cfaa979d39cd851aef" is_pre_election: true
I20260928 21:53:17.238927 22662 raft_consensus.cc:2454] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d61032784d2d4d69a0b8f062d0b47d23 in term 0.
I20260928 21:53:17.238989 22528 raft_consensus.cc:2454] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d61032784d2d4d69a0b8f062d0b47d23 in term 0.
I20260928 21:53:17.239163 22330 leader_election.cc:302] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580242de72764f9caaef4e698b7c1c3e, d61032784d2d4d69a0b8f062d0b47d23; no voters:
I20260928 21:53:17.239315 22728 raft_consensus.cc:2781] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:53:17.239373 22728 raft_consensus.cc:482] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:53:17.239388 22728 raft_consensus.cc:3037] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:17.240048 22728 raft_consensus.cc:504] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 } } peers { permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 } } peers { permanent_uuid: "d61032784d2d4d69a0b8f062d0b47d23" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44201 } }
I20260928 21:53:17.240161 22728 leader_election.cc:288] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [CANDIDATE]: Term 1 election: Requested vote from peers bdee04250b2c42cfaa979d39cd851aef (127.3.0.194:42841), 580242de72764f9caaef4e698b7c1c3e (127.3.0.195:41189)
I20260928 21:53:17.240332 22528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0123b161ca3643bface36d34282ab698" candidate_uuid: "d61032784d2d4d69a0b8f062d0b47d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdee04250b2c42cfaa979d39cd851aef"
I20260928 21:53:17.240404 22528 raft_consensus.cc:3037] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:17.240536 22662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0123b161ca3643bface36d34282ab698" candidate_uuid: "d61032784d2d4d69a0b8f062d0b47d23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580242de72764f9caaef4e698b7c1c3e"
I20260928 21:53:17.240594 22662 raft_consensus.cc:3037] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:17.241088 22528 raft_consensus.cc:2454] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d61032784d2d4d69a0b8f062d0b47d23 in term 1.
I20260928 21:53:17.241218 22662 raft_consensus.cc:2454] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d61032784d2d4d69a0b8f062d0b47d23 in term 1.
I20260928 21:53:17.241252 22328 leader_election.cc:302] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdee04250b2c42cfaa979d39cd851aef, d61032784d2d4d69a0b8f062d0b47d23; no voters:
I20260928 21:53:17.241417 22728 raft_consensus.cc:2781] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:17.241564 22728 raft_consensus.cc:686] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 1 LEADER]: Becoming Leader. State: Replica: d61032784d2d4d69a0b8f062d0b47d23, State: Running, Role: LEADER
I20260928 21:53:17.241637 22728 consensus_queue.cc:237] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 } } peers { permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 } } peers { permanent_uuid: "d61032784d2d4d69a0b8f062d0b47d23" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44201 } }
I20260928 21:53:17.242420 22249 catalog_manager.cc:5495] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 reported cstate change: term changed from 0 to 1, leader changed from <none> to d61032784d2d4d69a0b8f062d0b47d23 (127.3.0.193). New cstate: current_term: 1 leader_uuid: "d61032784d2d4d69a0b8f062d0b47d23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d61032784d2d4d69a0b8f062d0b47d23" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 44201 } health_report { overall_health: HEALTHY } } }
I20260928 21:53:17.291611 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.289417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.3.0.193:44201)
0928 21:53:17.289464 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.289466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.289474 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:53:17.289511 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.289511 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.289531 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.289587 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:17.289684 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.290142 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.290880 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.291072 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.291133 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.291334 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:17.291440 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:17.291505 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.291534 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.291552 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:53:17.293408 22528 raft_consensus.cc:1261] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef [term 1 FOLLOWER]: Refusing update from remote peer d61032784d2d4d69a0b8f062d0b47d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:17.293416 22662 raft_consensus.cc:1261] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e [term 1 FOLLOWER]: Refusing update from remote peer d61032784d2d4d69a0b8f062d0b47d23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:17.293720 22728 consensus_queue.cc:1035] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdee04250b2c42cfaa979d39cd851aef" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 42841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:17.293856 22733 consensus_queue.cc:1035] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580242de72764f9caaef4e698b7c1c3e" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:17.296605 22737 mvcc.cc:205] Tried to move back new op lower bound from 7334430299311185920 to 7334430299102261248. Current Snapshot: MvccSnapshot[applied={T|T < 7334430299311185920}]
I20260928 21:53:17.298292 22249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:17.298386 22249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:17.299019 22249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b768994b2f84b2ab09ab1a231e10459: Sending DeleteTablet for 3 replicas of tablet 0123b161ca3643bface36d34282ab698
I20260928 21:53:17.299347 22508 tablet_service.cc:1514] Processing DeleteTablet for tablet 0123b161ca3643bface36d34282ab698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:17 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:42186
I20260928 21:53:17.299360 22374 tablet_service.cc:1514] Processing DeleteTablet for tablet 0123b161ca3643bface36d34282ab698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:17 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:53266
I20260928 21:53:17.299607 22747 tablet_replica.cc:330] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23: stopping tablet replica
I20260928 21:53:17.299588 22642 tablet_service.cc:1514] Processing DeleteTablet for tablet 0123b161ca3643bface36d34282ab698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:17 UTC) from {username='kudu', principal='kudu/127.3.0.254@KRBTEST.COM'} at 127.0.0.1:51104
I20260928 21:53:17.299758 22747 raft_consensus.cc:2229] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:53:17.299805 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 22311
I20260928 21:53:17.299923 22747 raft_consensus.cc:2258] T 0123b161ca3643bface36d34282ab698 P d61032784d2d4d69a0b8f062d0b47d23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:17.300881 22746 tablet_replica.cc:330] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: stopping tablet replica
I20260928 21:53:17.300976 22746 raft_consensus.cc:2229] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:17.301083 22746 raft_consensus.cc:2258] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:17.301474 22746 ts_tablet_manager.cc:1905] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:17.302085 22748 tablet_replica.cc:330] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: stopping tablet replica
I20260928 21:53:17.302184 22748 raft_consensus.cc:2229] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:17.302276 22748 raft_consensus.cc:2258] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:17.302677 22748 ts_tablet_manager.cc:1905] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:17.302938 22746 ts_tablet_manager.cc:1918] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:17.303010 22746 log.cc:1196] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/0123b161ca3643bface36d34282ab698
I20260928 21:53:17.303222 22746 ts_tablet_manager.cc:1939] T 0123b161ca3643bface36d34282ab698 P bdee04250b2c42cfaa979d39cd851aef: Deleting consensus metadata
I20260928 21:53:17.303664 22234 catalog_manager.cc:4841] TS bdee04250b2c42cfaa979d39cd851aef (127.3.0.194:42841): tablet 0123b161ca3643bface36d34282ab698 (table test-table [id=9dadaaf40c4e4696be8f24fe0cabe044]) successfully deleted
I20260928 21:53:17.304224 22748 ts_tablet_manager.cc:1918] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:17.304289 22748 log.cc:1196] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/0123b161ca3643bface36d34282ab698
I20260928 21:53:17.304543 22748 ts_tablet_manager.cc:1939] T 0123b161ca3643bface36d34282ab698 P 580242de72764f9caaef4e698b7c1c3e: Deleting consensus metadata
I20260928 21:53:17.304972 22236 catalog_manager.cc:4841] TS 580242de72764f9caaef4e698b7c1c3e (127.3.0.195:41189): tablet 0123b161ca3643bface36d34282ab698 (table test-table [id=9dadaaf40c4e4696be8f24fe0cabe044]) successfully deleted
W20260928 21:53:17.305128 22236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:44201 (error 108)
I20260928 21:53:17.305321 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 22445
I20260928 21:53:17.305984 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.305506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:44201 (local address 127.0.0.1:53284)
0928 21:53:17.305911 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.305951 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
W20260928 21:53:17.306416 22236 catalog_manager.cc:4584] TS d61032784d2d4d69a0b8f062d0b47d23 (127.3.0.193:44201): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0123b161ca3643bface36d34282ab698: Network error: Client connection negotiation failed: client connection to 127.3.0.193:44201: connect: Connection refused (error 111)
I20260928 21:53:17.310153 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 22579
I20260928 21:53:17.313954 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 22217
2026-09-28T21:53:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2134 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T21:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:17Z Disabled control of system clock
I20260928 21:53:17.338987 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38401
--webserver_interface=127.3.0.254
--webserver_port=0
--builtin_ntp_servers=127.3.0.212:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:38401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:17.417546 22753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:17.417768 22753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:17.417788 22753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:17.418944 22753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:17.418989 22753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:17.419001 22753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:17.419013 22753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:17.419023 22753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:17.420357 22753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:38401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:38401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:17.420614 22753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:17.420899 22753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:17.425400 22758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:17.425417 22761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:17.425436 22753 server_base.cc:1034] running on GCE node
W20260928 21:53:17.425477 22759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:17.425953 22753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:17.426246 22753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:17.427390 22753 hybrid_clock.cc:648] HybridClock initialized: now 1790632397427374 us; error 35 us; skew 500 ppm
I20260928 21:53:17.428659 22753 webserver.cc:460] Webserver started at http://127.3.0.254:43131/ using document root <none> and password file <none>
I20260928 21:53:17.428885 22753 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:17.428951 22753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:17.429075 22753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:17.430128 22753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "b6b2f8def7ce413e8c0fb35e43406857"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "b049c018e411ac748c0e52a4fef090dd"
server_key_iv: "823445e994eaffdf618ca9653382d3d8"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.430460 22753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "b6b2f8def7ce413e8c0fb35e43406857"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "b049c018e411ac748c0e52a4fef090dd"
server_key_iv: "823445e994eaffdf618ca9653382d3d8"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.431658 22753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:17.432294 22767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.432471 22753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:17.432554 22753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "b6b2f8def7ce413e8c0fb35e43406857"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "b049c018e411ac748c0e52a4fef090dd"
server_key_iv: "823445e994eaffdf618ca9653382d3d8"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.432629 22753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:17.446403 22753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:17.446715 22753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:17.446842 22753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:17.450606 22753 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:38401
I20260928 21:53:17.450637 22819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:38401 every 8 connection(s)
I20260928 21:53:17.450974 22753 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:17.451860 22820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:17.452901 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 22753
I20260928 21:53:17.452996 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:17.453162 3075 external_mini_cluster.cc:1428] Setting key 9a63ea32ce3b865ea624788ed4dabaf7
I20260928 21:53:17.454442 22820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857: Bootstrap starting.
I20260928 21:53:17.455088 22820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:17.455343 22820 log.cc:824] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:17.456058 22820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857: No bootstrap required, opened a new log
I20260928 21:53:17.458168 22820 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b2f8def7ce413e8c0fb35e43406857" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38401 } }
I20260928 21:53:17.458302 22820 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:17.458333 22820 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6b2f8def7ce413e8c0fb35e43406857, State: Initialized, Role: FOLLOWER
I20260928 21:53:17.458487 22820 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b2f8def7ce413e8c0fb35e43406857" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38401 } }
I20260928 21:53:17.458567 22820 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:17.458598 22820 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:17.458660 22820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:17.458822 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.453667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.3.0.254:38401)
0928 21:53:17.453855 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.453863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.453887 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:53:17.454304 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.454311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.454373 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.454502 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:17.454720 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.455755 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.456835 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.457275 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.457372 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.458041 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:17.458045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:17.458045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:17.458204 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:17.458256 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:17.458267 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:17.458321 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.458434 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.458519 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260928 21:53:17.459295 22820 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b2f8def7ce413e8c0fb35e43406857" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38401 } }
I20260928 21:53:17.459395 22820 leader_election.cc:302] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6b2f8def7ce413e8c0fb35e43406857; no voters:
I20260928 21:53:17.459545 22820 leader_election.cc:288] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:17.459586 22825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:17.459692 22825 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [term 1 LEADER]: Becoming Leader. State: Replica: b6b2f8def7ce413e8c0fb35e43406857, State: Running, Role: LEADER
I20260928 21:53:17.459767 22825 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b2f8def7ce413e8c0fb35e43406857" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38401 } }
I20260928 21:53:17.459797 22820 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:17.460072 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6b2f8def7ce413e8c0fb35e43406857. Latest consensus state: current_term: 1 leader_uuid: "b6b2f8def7ce413e8c0fb35e43406857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b2f8def7ce413e8c0fb35e43406857" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38401 } } }
I20260928 21:53:17.460136 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:17.460100 22827 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6b2f8def7ce413e8c0fb35e43406857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6b2f8def7ce413e8c0fb35e43406857" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 38401 } } }
I20260928 21:53:17.460177 22827 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:17.460418 22830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:17.460991 22830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:17.462435 22830 catalog_manager.cc:1347] Generated new cluster ID: e03eb20a9d7c4384b620b5efd91c870f
I20260928 21:53:17.462488 22830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:17.510177 22830 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:17.510727 22830 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:17.517547 22830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857: Generated new TSK 0
I20260928 21:53:17.517731 22830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:17.529930 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.193:0
--local_ip_for_outbound_sockets=127.3.0.193
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38401
--builtin_ntp_servers=127.3.0.212:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:17.630023 22844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:17.630223 22844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:17.630244 22844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:17.631397 22844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:17.631434 22844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:17.631458 22844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.193
I20260928 21:53:17.632809 22844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.0.193
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.193
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:17.633041 22844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:17.633322 22844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:17.638247 22852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:17.638262 22850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:17.638245 22849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:17.638474 22844 server_base.cc:1034] running on GCE node
I20260928 21:53:17.638720 22844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:17.638973 22844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:17.640142 22844 hybrid_clock.cc:648] HybridClock initialized: now 1790632397640122 us; error 31 us; skew 500 ppm
I20260928 21:53:17.641357 22844 webserver.cc:460] Webserver started at http://127.3.0.193:34475/ using document root <none> and password file <none>
I20260928 21:53:17.641582 22844 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:17.641629 22844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:17.641757 22844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:17.642872 22844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/data/instance:
uuid: "b3aa64977d7c42ae836f332044c6cc27"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "64fc9499c8dc20d82224916db0d38c5f"
server_key_iv: "df9dcb26cc58685bdc8614389c6dd252"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.643183 22844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance:
uuid: "b3aa64977d7c42ae836f332044c6cc27"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "64fc9499c8dc20d82224916db0d38c5f"
server_key_iv: "df9dcb26cc58685bdc8614389c6dd252"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.644348 22844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:17.645063 22858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.645210 22844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.645293 22844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/wal
uuid: "b3aa64977d7c42ae836f332044c6cc27"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "64fc9499c8dc20d82224916db0d38c5f"
server_key_iv: "df9dcb26cc58685bdc8614389c6dd252"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.645363 22844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:17.660269 22844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:17.660650 22844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:17.660808 22844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:17.661041 22844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:17.661381 22844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:17.661425 22844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.661455 22844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:17.661478 22844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.666932 22844 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.193:45529
I20260928 21:53:17.667004 22971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.193:45529 every 8 connection(s)
I20260928 21:53:17.667304 22844 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/data/info.pb
I20260928 21:53:17.672096 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.667985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.193:54403 (local address 127.3.0.254:38401)
0928 21:53:17.668040 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.668042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.668294 (+ 252us) server_negotiation.cc:407] Connection header received
0928 21:53:17.668817 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.668818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.668830 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.668879 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:17.669517 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.670005 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.670926 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.671106 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.671149 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.671877 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:17.671878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:17.671879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:17.671884 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:17.671899 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:17.671903 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:17.671939 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.672003 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.672034 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:53:17.672304 22974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.667998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38401 (local address 127.3.0.193:54403)
0928 21:53:17.668171 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.668204 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:17.668725 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:17.668937 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:17.668940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:17.668949 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:17.669456 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.669461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.670067 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.670068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.670865 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.670867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.671161 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.671161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.671286 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.671748 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:53:17.671749 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:17.671781 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:53:17.671804 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:17.671946 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:17.671949 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:17.671951 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:53:17.672001 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:53:17.672084 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260928 21:53:17.672776 22972 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38401
I20260928 21:53:17.672861 22972 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:17.673010 22972 heartbeater.cc:511] Master 127.3.0.254:38401 requested a full tablet report, sending...
I20260928 21:53:17.673466 22784 ts_manager.cc:194] Registered new tserver with Master: b3aa64977d7c42ae836f332044c6cc27 (127.3.0.193:45529)
I20260928 21:53:17.674053 22784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.193:54403
I20260928 21:53:17.675443 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 22844
I20260928 21:53:17.675536 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-0/wal/instance
I20260928 21:53:17.675699 3075 external_mini_cluster.cc:1428] Setting key 4ed6beb3e2f60af2080ebb479af9a675
I20260928 21:53:17.676544 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.194:0
--local_ip_for_outbound_sockets=127.3.0.194
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38401
--builtin_ntp_servers=127.3.0.212:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:53:17.755767 22975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:17.755973 22975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:17.755993 22975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:17.757135 22975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:17.757174 22975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:17.757197 22975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.194
I20260928 21:53:17.758507 22975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.0.194
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.194
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:17.758715 22975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:17.758997 22975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:17.763705 22981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:17.763759 22983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:17.763763 22980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:17.764029 22975 server_base.cc:1034] running on GCE node
I20260928 21:53:17.764222 22975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:17.764473 22975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:17.765621 22975 hybrid_clock.cc:648] HybridClock initialized: now 1790632397765607 us; error 29 us; skew 500 ppm
I20260928 21:53:17.766865 22975 webserver.cc:460] Webserver started at http://127.3.0.194:35253/ using document root <none> and password file <none>
I20260928 21:53:17.767091 22975 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:17.767134 22975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:17.767261 22975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:17.768307 22975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/data/instance:
uuid: "55440eb988dd47168984bda7158c3bed"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "5d98c9d5d811d2dff92ee8ca248ebc65"
server_key_iv: "75cca94a80e0e59e9ea43ea7137a1e4f"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.768652 22975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance:
uuid: "55440eb988dd47168984bda7158c3bed"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "5d98c9d5d811d2dff92ee8ca248ebc65"
server_key_iv: "75cca94a80e0e59e9ea43ea7137a1e4f"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.769763 22975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:53:17.770376 22989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.770531 22975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:53:17.770601 22975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/wal
uuid: "55440eb988dd47168984bda7158c3bed"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "5d98c9d5d811d2dff92ee8ca248ebc65"
server_key_iv: "75cca94a80e0e59e9ea43ea7137a1e4f"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.770668 22975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:17.789158 22975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:17.789458 22975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:17.789575 22975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:17.789806 22975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:17.790130 22975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:17.790162 22975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.790184 22975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:17.790206 22975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.795480 22975 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.194:46021
I20260928 21:53:17.795549 23102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.194:46021 every 8 connection(s)
I20260928 21:53:17.795833 22975 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/data/info.pb
I20260928 21:53:17.800865 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.796560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.194:38649 (local address 127.3.0.254:38401)
0928 21:53:17.796603 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.796605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.796841 (+ 236us) server_negotiation.cc:407] Connection header received
0928 21:53:17.797345 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.797346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.797383 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.797437 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:17.798109 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.798604 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.799582 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.799771 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.799824 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.800613 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:17.800614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:17.800614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:17.800619 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:17.800636 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:17.800640 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:17.800680 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.800754 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.800782 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:53:17.800916 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 22975
I20260928 21:53:17.801014 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/wal/instance
I20260928 21:53:17.801110 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.796535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38401 (local address 127.3.0.194:38649)
0928 21:53:17.796722 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.796757 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:17.797252 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:17.797519 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:17.797523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:17.797534 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:17.798024 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.798030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.798689 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.798690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.799503 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.799505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.799842 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.799843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.799970 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.800484 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:53:17.800484 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:17.800517 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:53:17.800540 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:17.800686 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:17.800692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:17.800695 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:17.800754 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:53:17.800890 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260928 21:53:17.801226 3075 external_mini_cluster.cc:1428] Setting key 77b2e3fff23bf8f5d304c2e00ea4964f
I20260928 21:53:17.801581 23103 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38401
I20260928 21:53:17.801671 23103 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:17.801862 23103 heartbeater.cc:511] Master 127.3.0.254:38401 requested a full tablet report, sending...
I20260928 21:53:17.802124 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.0.195:0
--local_ip_for_outbound_sockets=127.3.0.195
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38401
--builtin_ntp_servers=127.3.0.212:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:53:17.802222 22784 ts_manager.cc:194] Registered new tserver with Master: 55440eb988dd47168984bda7158c3bed (127.3.0.194:46021)
I20260928 21:53:17.802639 22784 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.194:38649
W20260928 21:53:17.882032 23106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:17.882251 23106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:17.882273 23106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:17.883414 23106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:17.883450 23106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:17.883473 23106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.0.195
I20260928 21:53:17.884805 23106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.0.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.0.195
--webserver_port=0
--tserver_master_addrs=127.3.0.254:38401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.0.195
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:17.885010 23106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:17.885263 23106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:17.890196 23112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:17.890215 23111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:17.890280 23106 server_base.cc:1034] running on GCE node
W20260928 21:53:17.890213 23114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:17.890585 23106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:17.890833 23106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:17.892047 23106 hybrid_clock.cc:648] HybridClock initialized: now 1790632397892033 us; error 32 us; skew 500 ppm
I20260928 21:53:17.893488 23106 webserver.cc:460] Webserver started at http://127.3.0.195:41155/ using document root <none> and password file <none>
I20260928 21:53:17.893723 23106 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:17.893783 23106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:17.893924 23106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:17.895001 23106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/data/instance:
uuid: "a4287c607161475d949652e903a05225"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "b80d09fd13512050c063317db60e476b"
server_key_iv: "f5362177f193859d4f3ccb907f0e0520"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.895342 23106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance:
uuid: "a4287c607161475d949652e903a05225"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "b80d09fd13512050c063317db60e476b"
server_key_iv: "f5362177f193859d4f3ccb907f0e0520"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.896646 23106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:17.897426 23120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.897600 23106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:17.897680 23106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/wal
uuid: "a4287c607161475d949652e903a05225"
format_stamp: "Formatted at 2026-09-28 21:53:17 on dist-test-slave-jrck"
server_key: "b80d09fd13512050c063317db60e476b"
server_key_iv: "f5362177f193859d4f3ccb907f0e0520"
server_key_version: "encryptionkey@0"
I20260928 21:53:17.897748 23106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:17.905167 23106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:17.905445 23106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:17.905565 23106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:17.905759 23106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:53:17.906042 23106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:53:17.906073 23106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.906100 23106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:53:17.906123 23106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:17.911446 23106 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.195:40189
I20260928 21:53:17.911484 23233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.195:40189 every 8 connection(s)
I20260928 21:53:17.911783 23106 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/data/info.pb
I20260928 21:53:17.916405 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.912507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:39349 (local address 127.3.0.254:38401)
0928 21:53:17.912562 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.912564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.912759 (+ 195us) server_negotiation.cc:407] Connection header received
0928 21:53:17.913219 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.913220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.913230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.913280 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:17.913900 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.914383 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.915306 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.915486 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.915528 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.916196 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:17.916197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:17.916197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:17.916201 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:17.916215 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:17.916220 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:17.916256 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.916318 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.916338 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:53:17.916615 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.912441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:38401 (local address 127.3.0.195:39349)
0928 21:53:17.912642 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.912676 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:17.913140 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:17.913334 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:17.913338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:17.913346 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:17.913833 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.913837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.914446 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.914447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.915248 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.915251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.915537 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.915538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.915636 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.916080 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:53:17.916080 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:17.916108 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:53:17.916130 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:17.916264 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:17.916266 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:17.916269 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:17.916315 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:53:17.916390 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260928 21:53:17.916668 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 23106
I20260928 21:53:17.916771 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/wal/instance
I20260928 21:53:17.916962 3075 external_mini_cluster.cc:1428] Setting key 922723d7397b0a7aea491b579c246d41
I20260928 21:53:17.917104 23234 heartbeater.cc:347] Connected to a master server at 127.3.0.254:38401
I20260928 21:53:17.917186 23234 heartbeater.cc:464] Registering TS with master...
I20260928 21:53:17.917366 23234 heartbeater.cc:511] Master 127.3.0.254:38401 requested a full tablet report, sending...
I20260928 21:53:17.917704 22783 ts_manager.cc:194] Registered new tserver with Master: a4287c607161475d949652e903a05225 (127.3.0.195:40189)
I20260928 21:53:17.918075 22783 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.0.195:39349
I20260928 21:53:17.928866 3075 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:53:17.932839 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.930193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.3.0.254:38401)
0928 21:53:17.930262 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.930264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.930333 (+ 69us) server_negotiation.cc:407] Connection header received
0928 21:53:17.930383 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.930384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.930393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.930449 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:17.930580 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.931065 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.931785 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.931997 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.932051 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.932654 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:17.932655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:17.932656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:17.932660 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:17.932673 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:17.932676 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:17.932716 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.932755 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.932779 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:53:17.934446 22783 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:53:17.936080 22783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:53:17.942201 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.938315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:45529 (local address 127.0.0.1:40826)
0928 21:53:17.938528 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.938537 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:17.938589 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:17.938876 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:17.938879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:17.938882 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:17.938952 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.938954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.940247 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.940247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.941304 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.941305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.941924 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.941925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.941997 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.942010 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 21:53:17.942072 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:53:17.942123 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":7,"thread_start_us":61,"threads_started":1}
I20260928 21:53:17.942210 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.938529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40826 (local address 127.3.0.193:45529)
0928 21:53:17.938716 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.938720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.938728 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:53:17.938760 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.938761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.938772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.938854 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:17.939038 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.940166 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.941380 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.941848 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.941914 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.942008 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.942085 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.942123 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260928 21:53:17.942400 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.938118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.195:40189 (local address 127.0.0.1:44188)
0928 21:53:17.938286 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.938330 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:17.938413 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:17.939149 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:17.939150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:17.939152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:17.939183 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.939184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.940347 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.940348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.941301 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.941304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.942136 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.942136 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.942235 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.942239 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:53:17.942272 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 21:53:17.942313 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260928 21:53:17.942651 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.938227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.3.0.194:46021)
0928 21:53:17.938705 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.938710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.938720 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:53:17.938759 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.938783 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.938795 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.938865 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:17.939099 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.939842 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.941212 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.941948 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.942017 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.942103 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.942535 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.942576 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":79,"threads_started":1}
I20260928 21:53:17.942809 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.938124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:46021 (local address 127.0.0.1:33646)
0928 21:53:17.938390 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.938398 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:17.938437 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:17.938969 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:17.938969 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:17.938971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:17.939023 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.939025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.939913 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.939914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.941134 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.941136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.942135 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.942136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.942221 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.942226 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:53:17.942407 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 21:53:17.942736 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":23,"thread_start_us":53,"threads_started":1}
I20260928 21:53:17.942821 22906 tablet_service.cc:1467] Processing CreateTablet for tablet 693f2d27d45348f2ae05eec1b73b391a (DEFAULT_TABLE table=test-table [id=c98b0501a97e4cf8a3b2653bf3b09f8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:17.943038 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.938278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44188 (local address 127.3.0.195:40189)
0928 21:53:17.938986 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.938990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.939000 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:53:17.939034 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.939036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.939049 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.939146 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:17.939236 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.940280 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.941388 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.942054 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.942797 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.942900 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.942927 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.942964 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":93,"threads_started":1}
I20260928 21:53:17.943351 22906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693f2d27d45348f2ae05eec1b73b391a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:17.943634 23168 tablet_service.cc:1467] Processing CreateTablet for tablet 693f2d27d45348f2ae05eec1b73b391a (DEFAULT_TABLE table=test-table [id=c98b0501a97e4cf8a3b2653bf3b09f8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:17.943671 23037 tablet_service.cc:1467] Processing CreateTablet for tablet 693f2d27d45348f2ae05eec1b73b391a (DEFAULT_TABLE table=test-table [id=c98b0501a97e4cf8a3b2653bf3b09f8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:53:17.943917 23168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693f2d27d45348f2ae05eec1b73b391a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:17.943996 23037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693f2d27d45348f2ae05eec1b73b391a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:17.946328 23248 tablet_bootstrap.cc:492] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: Bootstrap starting.
I20260928 21:53:17.946676 23249 tablet_bootstrap.cc:492] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27: Bootstrap starting.
I20260928 21:53:17.946981 23248 tablet_bootstrap.cc:654] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:17.947244 23248 log.cc:824] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:17.947297 23249 tablet_bootstrap.cc:654] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:17.947474 23250 tablet_bootstrap.cc:492] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: Bootstrap starting.
I20260928 21:53:17.947629 23249 log.cc:824] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:17.948012 23248 tablet_bootstrap.cc:492] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: No bootstrap required, opened a new log
I20260928 21:53:17.948056 23250 tablet_bootstrap.cc:654] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:17.948077 23248 ts_tablet_manager.cc:1397] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:53:17.948333 23250 log.cc:824] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:17.948596 23249 tablet_bootstrap.cc:492] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27: No bootstrap required, opened a new log
I20260928 21:53:17.948664 23249 ts_tablet_manager.cc:1397] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:53:17.948962 23250 tablet_bootstrap.cc:492] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: No bootstrap required, opened a new log
I20260928 21:53:17.949018 23250 ts_tablet_manager.cc:1397] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:53:17.950549 23248 raft_consensus.cc:348] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 } } peers { permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 } } peers { permanent_uuid: "a4287c607161475d949652e903a05225" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 40189 } }
I20260928 21:53:17.950722 23248 raft_consensus.cc:374] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:17.950755 23248 raft_consensus.cc:729] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4287c607161475d949652e903a05225, State: Initialized, Role: FOLLOWER
I20260928 21:53:17.950908 23248 consensus_queue.cc:260] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 } } peers { permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 } } peers { permanent_uuid: "a4287c607161475d949652e903a05225" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 40189 } }
I20260928 21:53:17.950929 23250 raft_consensus.cc:348] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 } } peers { permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 } } peers { permanent_uuid: "a4287c607161475d949652e903a05225" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 40189 } }
I20260928 21:53:17.951073 23250 raft_consensus.cc:374] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:17.951115 23250 raft_consensus.cc:729] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55440eb988dd47168984bda7158c3bed, State: Initialized, Role: FOLLOWER
I20260928 21:53:17.951032 23249 raft_consensus.cc:348] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 } } peers { permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 } } peers { permanent_uuid: "a4287c607161475d949652e903a05225" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 40189 } }
I20260928 21:53:17.951167 23249 raft_consensus.cc:374] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:17.951170 23248 ts_tablet_manager.cc:1428] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 21:53:17.951205 23249 raft_consensus.cc:729] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3aa64977d7c42ae836f332044c6cc27, State: Initialized, Role: FOLLOWER
I20260928 21:53:17.951226 23250 consensus_queue.cc:260] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 } } peers { permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 } } peers { permanent_uuid: "a4287c607161475d949652e903a05225" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 40189 } }
I20260928 21:53:17.951282 23234 heartbeater.cc:503] Master 127.3.0.254:38401 was elected leader, sending a full tablet report...
I20260928 21:53:17.951305 23249 consensus_queue.cc:260] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 } } peers { permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 } } peers { permanent_uuid: "a4287c607161475d949652e903a05225" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 40189 } }
I20260928 21:53:17.951476 23250 ts_tablet_manager.cc:1428] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 21:53:17.951599 23103 heartbeater.cc:503] Master 127.3.0.254:38401 was elected leader, sending a full tablet report...
I20260928 21:53:17.951545 23249 ts_tablet_manager.cc:1428] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 21:53:17.951824 22972 heartbeater.cc:503] Master 127.3.0.254:38401 was elected leader, sending a full tablet report...
I20260928 21:53:17.962153 23254 raft_consensus.cc:482] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:53:17.962258 23254 raft_consensus.cc:504] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 } } peers { permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 } } peers { permanent_uuid: "a4287c607161475d949652e903a05225" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 40189 } }
I20260928 21:53:17.962539 23254 leader_election.cc:288] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55440eb988dd47168984bda7158c3bed (127.3.0.194:46021), b3aa64977d7c42ae836f332044c6cc27 (127.3.0.193:45529)
I20260928 21:53:17.965319 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.962754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:55741 (local address 127.3.0.194:46021)
0928 21:53:17.962840 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.962842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.962850 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:53:17.962881 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.962881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.962891 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.962959 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:17.963092 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.963631 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.964650 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.965018 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.965069 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.965077 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.965212 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.965251 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 21:53:17.965304 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.962689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.194:46021 (local address 127.3.0.195:55741)
0928 21:53:17.962736 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.962747 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:17.962825 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:17.962963 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:17.962963 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:17.962965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:17.963031 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.963032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.963701 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.963702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.964575 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.964578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.965088 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.965089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.965151 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.965155 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:53:17.965198 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:53:17.965232 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 21:53:17.965380 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.962833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.0.195:45233 (local address 127.3.0.193:45529)
0928 21:53:17.962880 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.962881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.963024 (+ 143us) server_negotiation.cc:407] Connection header received
0928 21:53:17.963059 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.963060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.963072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.963121 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:53:17.963241 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.963705 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.964675 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.965047 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.965126 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.965135 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.965284 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.965322 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 21:53:17.965732 23057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693f2d27d45348f2ae05eec1b73b391a" candidate_uuid: "a4287c607161475d949652e903a05225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55440eb988dd47168984bda7158c3bed" is_pre_election: true
I20260928 21:53:17.965890 23057 raft_consensus.cc:2454] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4287c607161475d949652e903a05225 in term 0.
I20260928 21:53:17.965960 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.962777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:45529 (local address 127.3.0.195:45233)
0928 21:53:17.962953 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.962963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:17.963016 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:17.963123 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:17.963124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:17.963125 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:53:17.963183 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.963184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.963775 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.963775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.964595 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:17.964597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.965145 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.965146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:17.965216 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.965219 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:53:17.965264 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:53:17.965298 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260928 21:53:17.966096 23123 leader_election.cc:302] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55440eb988dd47168984bda7158c3bed, a4287c607161475d949652e903a05225; no voters:
I20260928 21:53:17.966197 23254 raft_consensus.cc:2781] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:53:17.966252 23254 raft_consensus.cc:482] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:53:17.966274 23254 raft_consensus.cc:3037] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:17.966378 22926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693f2d27d45348f2ae05eec1b73b391a" candidate_uuid: "a4287c607161475d949652e903a05225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3aa64977d7c42ae836f332044c6cc27" is_pre_election: true
I20260928 21:53:17.966509 22926 raft_consensus.cc:2454] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4287c607161475d949652e903a05225 in term 0.
I20260928 21:53:17.967036 23254 raft_consensus.cc:504] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 } } peers { permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 } } peers { permanent_uuid: "a4287c607161475d949652e903a05225" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 40189 } }
I20260928 21:53:17.967154 23254 leader_election.cc:288] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [CANDIDATE]: Term 1 election: Requested vote from peers 55440eb988dd47168984bda7158c3bed (127.3.0.194:46021), b3aa64977d7c42ae836f332044c6cc27 (127.3.0.193:45529)
I20260928 21:53:17.967295 23057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693f2d27d45348f2ae05eec1b73b391a" candidate_uuid: "a4287c607161475d949652e903a05225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55440eb988dd47168984bda7158c3bed"
I20260928 21:53:17.967310 22926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693f2d27d45348f2ae05eec1b73b391a" candidate_uuid: "a4287c607161475d949652e903a05225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3aa64977d7c42ae836f332044c6cc27"
I20260928 21:53:17.967377 22926 raft_consensus.cc:3037] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:17.967376 23057 raft_consensus.cc:3037] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:17.968225 23057 raft_consensus.cc:2454] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4287c607161475d949652e903a05225 in term 1.
I20260928 21:53:17.968225 22926 raft_consensus.cc:2454] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4287c607161475d949652e903a05225 in term 1.
I20260928 21:53:17.968391 23123 leader_election.cc:302] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55440eb988dd47168984bda7158c3bed, a4287c607161475d949652e903a05225; no voters:
I20260928 21:53:17.968497 23254 raft_consensus.cc:2781] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:17.968624 23254 raft_consensus.cc:686] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 1 LEADER]: Becoming Leader. State: Replica: a4287c607161475d949652e903a05225, State: Running, Role: LEADER
I20260928 21:53:17.968698 23254 consensus_queue.cc:237] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 } } peers { permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 } } peers { permanent_uuid: "a4287c607161475d949652e903a05225" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 40189 } }
I20260928 21:53:17.969502 22783 catalog_manager.cc:5495] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4287c607161475d949652e903a05225 (127.3.0.195). New cstate: current_term: 1 leader_uuid: "a4287c607161475d949652e903a05225" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4287c607161475d949652e903a05225" member_type: VOTER last_known_addr { host: "127.3.0.195" port: 40189 } health_report { overall_health: HEALTHY } } }
I20260928 21:53:17.977321 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.975072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.3.0.195:40189)
0928 21:53:17.975117 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:17.975119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:17.975128 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:53:17.975162 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:17.975163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:17.975178 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:17.975235 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:17.975388 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.975886 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.976668 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:17.976867 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:17.976930 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:17.977036 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:17.977140 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:17.977198 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:17.977228 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 21:53:17.977244 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:53:17.979012 23057 raft_consensus.cc:1261] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed [term 1 FOLLOWER]: Refusing update from remote peer a4287c607161475d949652e903a05225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:17.979065 22926 raft_consensus.cc:1261] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27 [term 1 FOLLOWER]: Refusing update from remote peer a4287c607161475d949652e903a05225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:17.979372 23254 consensus_queue.cc:1035] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3aa64977d7c42ae836f332044c6cc27" member_type: VOTER last_known_addr { host: "127.3.0.193" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:17.979494 23258 consensus_queue.cc:1035] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55440eb988dd47168984bda7158c3bed" member_type: VOTER last_known_addr { host: "127.3.0.194" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:17.981700 23261 mvcc.cc:205] Tried to move back new op lower bound from 7334430302119731200 to 7334430302080516096. Current Snapshot: MvccSnapshot[applied={T|T < 7334430302119731200}]
I20260928 21:53:17.982426 23262 mvcc.cc:205] Tried to move back new op lower bound from 7334430302119731200 to 7334430302080516096. Current Snapshot: MvccSnapshot[applied={T|T < 7334430302119731200}]
I20260928 21:53:17.983837 22783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:17.983922 22783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:53:17.984730 22783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6b2f8def7ce413e8c0fb35e43406857: Sending DeleteTablet for 3 replicas of tablet 693f2d27d45348f2ae05eec1b73b391a
I20260928 21:53:17.985080 23037 tablet_service.cc:1514] Processing DeleteTablet for tablet 693f2d27d45348f2ae05eec1b73b391a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:17 UTC) from {username='slave'} at 127.0.0.1:33646
I20260928 21:53:17.985083 23168 tablet_service.cc:1514] Processing DeleteTablet for tablet 693f2d27d45348f2ae05eec1b73b391a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:17 UTC) from {username='slave'} at 127.0.0.1:44188
I20260928 21:53:17.985293 23270 tablet_replica.cc:330] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: stopping tablet replica
I20260928 21:53:17.985296 22906 tablet_service.cc:1514] Processing DeleteTablet for tablet 693f2d27d45348f2ae05eec1b73b391a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:53:17 UTC) from {username='slave'} at 127.0.0.1:40826
I20260928 21:53:17.985388 23270 raft_consensus.cc:2229] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:17.985507 23270 raft_consensus.cc:2258] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:17.985522 23272 tablet_replica.cc:330] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27: stopping tablet replica
I20260928 21:53:17.985570 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 22844
I20260928 21:53:17.985705 23272 raft_consensus.cc:2229] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:53:17.985847 23272 raft_consensus.cc:2258] T 693f2d27d45348f2ae05eec1b73b391a P b3aa64977d7c42ae836f332044c6cc27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:17.987279 23270 ts_tablet_manager.cc:1905] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:17.988866 23270 ts_tablet_manager.cc:1918] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:17.988940 23270 log.cc:1196] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-1/wal/wals/693f2d27d45348f2ae05eec1b73b391a
I20260928 21:53:17.989164 23270 ts_tablet_manager.cc:1939] T 693f2d27d45348f2ae05eec1b73b391a P 55440eb988dd47168984bda7158c3bed: Deleting consensus metadata
I20260928 21:53:17.989562 22770 catalog_manager.cc:4841] TS 55440eb988dd47168984bda7158c3bed (127.3.0.194:46021): tablet 693f2d27d45348f2ae05eec1b73b391a (table test-table [id=c98b0501a97e4cf8a3b2653bf3b09f8f]) successfully deleted
W20260928 21:53:17.991266 22770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.0.193:45529 (error 108)
I20260928 21:53:17.991436 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 22975
I20260928 21:53:17.992089 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:17.991979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.193:45529 (local address 127.0.0.1:40828)
0928 21:53:17.992026 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:17.992059 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.193:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 21:53:17.992201 22770 catalog_manager.cc:4584] TS b3aa64977d7c42ae836f332044c6cc27 (127.3.0.193:45529): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 693f2d27d45348f2ae05eec1b73b391a: Network error: Client connection negotiation failed: client connection to 127.3.0.193:45529: connect: Connection refused (error 111)
I20260928 21:53:17.992735 23271 tablet_replica.cc:330] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: stopping tablet replica
I20260928 21:53:17.992836 23271 raft_consensus.cc:2229] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:53:17.992926 23271 raft_consensus.cc:2258] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:53:17.993281 23271 ts_tablet_manager.cc:1905] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:53:17.994608 23271 ts_tablet_manager.cc:1918] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:53:17.994674 23271 log.cc:1196] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: Deleting WAL directory at /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790632139349757-3075-0/minicluster-data/ts-2/wal/wals/693f2d27d45348f2ae05eec1b73b391a
I20260928 21:53:17.994890 23271 ts_tablet_manager.cc:1939] T 693f2d27d45348f2ae05eec1b73b391a P a4287c607161475d949652e903a05225: Deleting consensus metadata
I20260928 21:53:17.995270 22769 catalog_manager.cc:4841] TS a4287c607161475d949652e903a05225 (127.3.0.195:40189): tablet 693f2d27d45348f2ae05eec1b73b391a (table test-table [id=c98b0501a97e4cf8a3b2653bf3b09f8f]) successfully deleted
I20260928 21:53:17.997496 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 23106
I20260928 21:53:18.002364 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 22753
2026-09-28T21:53:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (687 ms)
[----------] 6 tests from EncryptionPolicyTest (8189 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:18 dist-test-slave-jrck krb5kdc[23276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:18 dist-test-slave-jrck krb5kdc[23276](info): set up 2 sockets
Sep 28 21:53:18 dist-test-slave-jrck krb5kdc[23276](info): commencing operation
krb5kdc: starting...
W20260928 21:53:18.956161 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:19 dist-test-slave-jrck krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:19.068995 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:19.148096 23289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:19.148307 23289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:19.148331 23289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:19.149489 23289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:19.149529 23289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:19.149544 23289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:19.149554 23289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:19.149564 23289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:19.150988 23289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42029
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:19.151193 23289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:19.151367 23289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:19.155953 23295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:19.155946 23294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:19.156150 23289 server_base.cc:1034] running on GCE node
W20260928 21:53:19.156236 23297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:19.156510 23289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:19.156785 23289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:19.157930 23289 hybrid_clock.cc:648] HybridClock initialized: now 1790632399157918 us; error 30 us; skew 500 ppm
Sep 28 21:53:19 dist-test-slave-jrck krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:19.160384 23289 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:19.160830 23289 webserver.cc:460] Webserver started at http://127.0.0.1:46281/ using document root <none> and password file <none>
I20260928 21:53:19.161051 23289 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:19.161100 23289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:19.161227 23289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:19.162279 23289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "0720506dfee54054a4b48b34fbff78b9"
format_stamp: "Formatted at 2026-09-28 21:53:19 on dist-test-slave-jrck"
server_key: "d4d1214226fa1c315444525a06619d28"
server_key_iv: "ab957cd6b9fa95b9434db94286de1f6c"
server_key_version: "encryptionkey@0"
I20260928 21:53:19.162618 23289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "0720506dfee54054a4b48b34fbff78b9"
format_stamp: "Formatted at 2026-09-28 21:53:19 on dist-test-slave-jrck"
server_key: "d4d1214226fa1c315444525a06619d28"
server_key_iv: "ab957cd6b9fa95b9434db94286de1f6c"
server_key_version: "encryptionkey@0"
I20260928 21:53:19.163906 23289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:53:19.164608 23304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:19.164757 23289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:19.164839 23289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "0720506dfee54054a4b48b34fbff78b9"
format_stamp: "Formatted at 2026-09-28 21:53:19 on dist-test-slave-jrck"
server_key: "d4d1214226fa1c315444525a06619d28"
server_key_iv: "ab957cd6b9fa95b9434db94286de1f6c"
server_key_version: "encryptionkey@0"
I20260928 21:53:19.164906 23289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:19.180967 23289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:19.183111 23289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:19.183251 23289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:19.186980 23289 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42029
I20260928 21:53:19.187026 23356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42029 every 8 connection(s)
I20260928 21:53:19.187358 23289 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:19.188349 23357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:19.190647 23357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9: Bootstrap starting.
I20260928 21:53:19.191206 23357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:19.191463 23357 log.cc:824] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:19.192209 23357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9: No bootstrap required, opened a new log
I20260928 21:53:19.193739 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 23289
I20260928 21:53:19.193881 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:19.193866 23357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0720506dfee54054a4b48b34fbff78b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42029 } }
I20260928 21:53:19.194036 23357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:19.194056 3075 external_mini_cluster.cc:1428] Setting key fefb0b680cd0361b7e6e78702c4bb702
I20260928 21:53:19.194072 23357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0720506dfee54054a4b48b34fbff78b9, State: Initialized, Role: FOLLOWER
I20260928 21:53:19.194216 23357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0720506dfee54054a4b48b34fbff78b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42029 } }
I20260928 21:53:19.194295 23357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:19.194326 23357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:19.194393 23357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:19.195372 23357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0720506dfee54054a4b48b34fbff78b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42029 } }
I20260928 21:53:19.195490 23357 leader_election.cc:302] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0720506dfee54054a4b48b34fbff78b9; no voters:
I20260928 21:53:19.195653 23357 leader_election.cc:288] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:19.195683 23362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:19.195812 23362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [term 1 LEADER]: Becoming Leader. State: Replica: 0720506dfee54054a4b48b34fbff78b9, State: Running, Role: LEADER
I20260928 21:53:19.195922 23357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:19.195894 23362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0720506dfee54054a4b48b34fbff78b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42029 } }
I20260928 21:53:19.196205 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0720506dfee54054a4b48b34fbff78b9. Latest consensus state: current_term: 1 leader_uuid: "0720506dfee54054a4b48b34fbff78b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0720506dfee54054a4b48b34fbff78b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42029 } } }
I20260928 21:53:19.196260 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:19.196400 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0720506dfee54054a4b48b34fbff78b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0720506dfee54054a4b48b34fbff78b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42029 } } }
I20260928 21:53:19.196445 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:19.196561 23367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:19.197000 23367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:19.198621 23367 catalog_manager.cc:1347] Generated new cluster ID: c9ade9f937e9473cb00d53f4431fe812
I20260928 21:53:19.198675 23367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:53:19 dist-test-slave-jrck krb5kdc[23276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:53:19.204634 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:19.194515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42265 (local address 127.0.0.1:42029)
0928 21:53:19.194716 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:19.194724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:19.194751 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:53:19.194850 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:19.194856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:19.194907 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:19.195019 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:19.195824 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:19.197211 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:19.198146 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:19.198587 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:19.198687 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:19.201226 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:19.201230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:19.201230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:19.201243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:19.202837 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:19.203228 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:19.203231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:19.203232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:19.203340 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:19.203555 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:19.203556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:19.203556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:19.203793 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:19.204309 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:19.204377 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 21:53:19.204480 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260928 21:53:19.205550 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:19.206130 23367 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:19.206776 23367 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:19.214627 23367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0720506dfee54054a4b48b34fbff78b9: Generated new TSK 0
I20260928 21:53:19.214871 23367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:19.219456 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:19.212874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38945 (local address 127.0.0.1:42029)
0928 21:53:19.212922 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:19.212924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:19.213098 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:53:19.213160 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:19.213161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:19.213172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:19.213217 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:19.213763 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:19.214359 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:19.215252 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:19.215465 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:19.215520 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:19.216807 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:19.216808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:19.216808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:19.216817 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:19.218301 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:19.218693 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:19.218695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:19.218695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:19.218754 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:19.218955 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:19.218956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:19.218956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:19.219070 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:19.219161 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:19.219296 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:53:19.219362 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:53:19.221544 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 23289
W20260928 21:53:19.225620 23274 connection.cc:535] client connection to 127.0.0.1:42029 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42029): Connection reset by peer (error 104)
2026-09-28T21:53:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:19 dist-test-slave-jrck krb5kdc[23388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:19 dist-test-slave-jrck krb5kdc[23388](info): set up 2 sockets
Sep 28 21:53:19 dist-test-slave-jrck krb5kdc[23388](info): commencing operation
krb5kdc: starting...
W20260928 21:53:20.160212 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:20 dist-test-slave-jrck krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:20.277156 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:20.356655 23401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:20.356881 23401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:20.356905 23401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:20.358069 23401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:20.358109 23401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:20.358122 23401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:20.358135 23401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:20.358158 23401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:20.359637 23401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:20.359880 23401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:20.360085 23401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:20.364698 23407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:20.364691 23406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:20.364785 23401 server_base.cc:1034] running on GCE node
W20260928 21:53:20.364691 23409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:20.365109 23401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:20.365402 23401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:20.366544 23401 hybrid_clock.cc:648] HybridClock initialized: now 1790632400366525 us; error 31 us; skew 500 ppm
Sep 28 21:53:20 dist-test-slave-jrck krb5kdc[23388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:20.372392 23401 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:20.372931 23401 webserver.cc:460] Webserver started at http://127.0.0.1:46075/ using document root <none> and password file <none>
I20260928 21:53:20.373142 23401 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:20.373189 23401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:20.373317 23401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:20.374385 23401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "5931abab42d94dff80c98c8ae1abc2b8"
format_stamp: "Formatted at 2026-09-28 21:53:20 on dist-test-slave-jrck"
server_key: "5b1ba715fa3c9e45260f2e5bf6f9b5f1"
server_key_iv: "0d57b46148eeeef076fd8093e514f9e6"
server_key_version: "encryptionkey@0"
I20260928 21:53:20.374740 23401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "5931abab42d94dff80c98c8ae1abc2b8"
format_stamp: "Formatted at 2026-09-28 21:53:20 on dist-test-slave-jrck"
server_key: "5b1ba715fa3c9e45260f2e5bf6f9b5f1"
server_key_iv: "0d57b46148eeeef076fd8093e514f9e6"
server_key_version: "encryptionkey@0"
I20260928 21:53:20.375973 23401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:20.376631 23416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:20.376845 23401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:20.376942 23401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "5931abab42d94dff80c98c8ae1abc2b8"
format_stamp: "Formatted at 2026-09-28 21:53:20 on dist-test-slave-jrck"
server_key: "5b1ba715fa3c9e45260f2e5bf6f9b5f1"
server_key_iv: "0d57b46148eeeef076fd8093e514f9e6"
server_key_version: "encryptionkey@0"
I20260928 21:53:20.377005 23401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:20.387106 23401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:20.388891 23401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:20.389026 23401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:20.392911 23401 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44453
I20260928 21:53:20.392947 23468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44453 every 8 connection(s)
I20260928 21:53:20.393278 23401 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:20.394237 23469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:20.396528 23469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8: Bootstrap starting.
I20260928 21:53:20.397112 23469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:20.397349 23469 log.cc:824] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:20.398021 23469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8: No bootstrap required, opened a new log
I20260928 21:53:20.399567 23469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5931abab42d94dff80c98c8ae1abc2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } }
I20260928 21:53:20.399711 23469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:20.399734 23469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5931abab42d94dff80c98c8ae1abc2b8, State: Initialized, Role: FOLLOWER
I20260928 21:53:20.399848 23469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5931abab42d94dff80c98c8ae1abc2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } }
I20260928 21:53:20.399932 23469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:20.399964 23469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:20.399996 23469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:20.400532 23469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5931abab42d94dff80c98c8ae1abc2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } }
I20260928 21:53:20.400617 23469 leader_election.cc:302] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5931abab42d94dff80c98c8ae1abc2b8; no voters:
I20260928 21:53:20.400740 23469 leader_election.cc:288] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:20.400780 23472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:20.400961 23469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:20.400971 23472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [term 1 LEADER]: Becoming Leader. State: Replica: 5931abab42d94dff80c98c8ae1abc2b8, State: Running, Role: LEADER
I20260928 21:53:20.401074 23472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5931abab42d94dff80c98c8ae1abc2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } }
I20260928 21:53:20.401448 23474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5931abab42d94dff80c98c8ae1abc2b8. Latest consensus state: current_term: 1 leader_uuid: "5931abab42d94dff80c98c8ae1abc2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5931abab42d94dff80c98c8ae1abc2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } } }
I20260928 21:53:20.401526 23474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:20.401830 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 23401
I20260928 21:53:20.401723 23481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:20.401940 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:20.402139 3075 external_mini_cluster.cc:1428] Setting key 71318d3fd016b46f0c250471dcd39fdb
I20260928 21:53:20.402232 23473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5931abab42d94dff80c98c8ae1abc2b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5931abab42d94dff80c98c8ae1abc2b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44453 } } }
I20260928 21:53:20.402401 23481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:20.403025 23473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:20.404102 23481 catalog_manager.cc:1347] Generated new cluster ID: f907e8a191da42f598986946a082642f
I20260928 21:53:20.404153 23481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:20.407174 23491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:20.402806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55075 (local address 127.0.0.1:44453)
0928 21:53:20.402955 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:20.402965 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:20.402998 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:53:20.403204 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:20.403211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:20.403269 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:20.403413 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:20.403620 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:20.404665 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:20.405414 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:20.405871 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:20.405967 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:20.406592 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:20.406595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:20.406595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:20.406755 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:20.406787 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:20.406797 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:20.406863 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:20.406960 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 21:53:20.406989 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260928 21:53:20.407992 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:20.415494 23491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:20.412726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57203 (local address 127.0.0.1:44453)
0928 21:53:20.412760 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:20.412762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:20.412955 (+ 193us) server_negotiation.cc:407] Connection header received
0928 21:53:20.413009 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:20.413010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:20.413020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:20.413060 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:20.413195 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:20.413699 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:20.414384 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:20.414607 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:20.414659 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:20.415259 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:20.415260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:20.415261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:20.415265 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:20.415280 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:20.415284 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:20.415330 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:20.415402 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 21:53:20.415413 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:53:20.416507 23481 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:20.416965 23481 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:20.419576 23481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5931abab42d94dff80c98c8ae1abc2b8: Generated new TSK 0
I20260928 21:53:20.419721 23481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:20.427508 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 23401
W20260928 21:53:20.431171 23386 connection.cc:535] client connection to 127.0.0.1:44453 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44453): Connection reset by peer (error 104)
2026-09-28T21:53:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:20 dist-test-slave-jrck krb5kdc[23500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:20 dist-test-slave-jrck krb5kdc[23500](info): set up 2 sockets
Sep 28 21:53:20 dist-test-slave-jrck krb5kdc[23500](info): commencing operation
krb5kdc: starting...
W20260928 21:53:21.368355 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:21 dist-test-slave-jrck krb5kdc[23500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:21.483037 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:21.563496 23513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:21.563699 23513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:21.563722 23513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:21.564893 23513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:21.564934 23513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:21.564947 23513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:21.564957 23513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:21.564970 23513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:21.566409 23513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44349
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:21.566632 23513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:21.566844 23513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:21.571463 23519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:21.571456 23518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:21.571568 23513 server_base.cc:1034] running on GCE node
W20260928 21:53:21.571456 23521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:21.571882 23513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:21.572163 23513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:21.573299 23513 hybrid_clock.cc:648] HybridClock initialized: now 1790632401573285 us; error 27 us; skew 500 ppm
Sep 28 21:53:21 dist-test-slave-jrck krb5kdc[23500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:21.575767 23513 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:21.576260 23513 webserver.cc:460] Webserver started at http://127.0.0.1:40797/ using document root <none> and password file <none>
I20260928 21:53:21.576530 23513 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:21.576586 23513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:21.576718 23513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:21.577839 23513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "e3df25c0e3e5498aa93f5a384d9d862c"
format_stamp: "Formatted at 2026-09-28 21:53:21 on dist-test-slave-jrck"
server_key: "b1bafe053b39e024be5c53f164088435"
server_key_iv: "b50dc09943549b348787062b9129e349"
server_key_version: "encryptionkey@0"
I20260928 21:53:21.578238 23513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "e3df25c0e3e5498aa93f5a384d9d862c"
format_stamp: "Formatted at 2026-09-28 21:53:21 on dist-test-slave-jrck"
server_key: "b1bafe053b39e024be5c53f164088435"
server_key_iv: "b50dc09943549b348787062b9129e349"
server_key_version: "encryptionkey@0"
I20260928 21:53:21.579577 23513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:21.580315 23528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:21.580555 23513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:21.580641 23513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "e3df25c0e3e5498aa93f5a384d9d862c"
format_stamp: "Formatted at 2026-09-28 21:53:21 on dist-test-slave-jrck"
server_key: "b1bafe053b39e024be5c53f164088435"
server_key_iv: "b50dc09943549b348787062b9129e349"
server_key_version: "encryptionkey@0"
I20260928 21:53:21.580715 23513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:21.583984 23513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:21.586004 23513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:21.586133 23513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:21.589932 23513 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44349
I20260928 21:53:21.589965 23580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44349 every 8 connection(s)
I20260928 21:53:21.590283 23513 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:21.591217 23581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:21.593528 23581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c: Bootstrap starting.
I20260928 21:53:21.594069 23581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:21.594331 23581 log.cc:824] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:21.595052 23581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c: No bootstrap required, opened a new log
I20260928 21:53:21.596670 23581 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3df25c0e3e5498aa93f5a384d9d862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260928 21:53:21.596823 23581 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:21.596845 23581 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3df25c0e3e5498aa93f5a384d9d862c, State: Initialized, Role: FOLLOWER
I20260928 21:53:21.596980 23581 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3df25c0e3e5498aa93f5a384d9d862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260928 21:53:21.597043 23581 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:21.597061 23581 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:21.597088 23581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:21.597599 23581 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3df25c0e3e5498aa93f5a384d9d862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260928 21:53:21.597682 23581 leader_election.cc:302] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3df25c0e3e5498aa93f5a384d9d862c; no voters:
I20260928 21:53:21.597822 23581 leader_election.cc:288] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:21.597904 23584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:21.598063 23584 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [term 1 LEADER]: Becoming Leader. State: Replica: e3df25c0e3e5498aa93f5a384d9d862c, State: Running, Role: LEADER
I20260928 21:53:21.598110 23581 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:21.598176 23584 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3df25c0e3e5498aa93f5a384d9d862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260928 21:53:21.598294 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 23513
I20260928 21:53:21.598423 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:21.598619 3075 external_mini_cluster.cc:1428] Setting key 9b90d42f1113ca0e947679db4e22ae1f
I20260928 21:53:21.598825 23585 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3df25c0e3e5498aa93f5a384d9d862c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3df25c0e3e5498aa93f5a384d9d862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } } }
I20260928 21:53:21.598832 23586 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3df25c0e3e5498aa93f5a384d9d862c. Latest consensus state: current_term: 1 leader_uuid: "e3df25c0e3e5498aa93f5a384d9d862c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3df25c0e3e5498aa93f5a384d9d862c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } } }
I20260928 21:53:21.599258 23586 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:21.599220 23585 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:21.599479 23596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:21.599996 23596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:21.601234 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:21.599252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46749 (local address 127.0.0.1:44349)
0928 21:53:21.599514 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:21.599564 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:21.599602 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:53:21.599745 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:21.599753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:21.599787 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:21.599946 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:21.600785 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:21.600790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:21.600790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:21.600795 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:21.600844 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:21.600859 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:21.600920 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:21.601029 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 21:53:21.601072 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":73,"threads_started":1}
I20260928 21:53:21.601878 23596 catalog_manager.cc:1347] Generated new cluster ID: 4a4cd2337f934fe3a9a743e4ea4bc403
I20260928 21:53:21.601930 23596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:21.602119 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:21.604617 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:21.603529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35609 (local address 127.0.0.1:44349)
0928 21:53:21.603561 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:21.603563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:21.603688 (+ 125us) server_negotiation.cc:407] Connection header received
0928 21:53:21.603757 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:21.603757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:21.603763 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:21.603810 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:21.604368 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:21.604369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:21.604369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:21.604373 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:21.604387 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:21.604391 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:21.604436 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:21.604535 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 21:53:21.604547 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:53:21.615056 23596 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:21.615237 23596 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:21.623471 23596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3df25c0e3e5498aa93f5a384d9d862c: Generated new TSK 0
I20260928 21:53:21.623663 23596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:21.630698 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 23513
2026-09-28T21:53:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:21 dist-test-slave-jrck krb5kdc[23612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:21 dist-test-slave-jrck krb5kdc[23612](info): set up 2 sockets
Sep 28 21:53:21 dist-test-slave-jrck krb5kdc[23612](info): commencing operation
krb5kdc: starting...
W20260928 21:53:22.580333 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:22 dist-test-slave-jrck krb5kdc[23612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:22.691152 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:22.771248 23625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:22.771481 23625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:22.771518 23625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:22.772782 23625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:22.772830 23625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:22.772850 23625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:22.772866 23625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:22.772879 23625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:22.774431 23625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:22.774703 23625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:22.774940 23625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:22.779837 23630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:22.779961 23625 server_base.cc:1034] running on GCE node
W20260928 21:53:22.780073 23631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:22.779896 23633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:22.780498 23625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:22.780791 23625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:22.781960 23625 hybrid_clock.cc:648] HybridClock initialized: now 1790632402781947 us; error 33 us; skew 500 ppm
Sep 28 21:53:22 dist-test-slave-jrck krb5kdc[23612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:22.784441 23625 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:22.784983 23625 webserver.cc:460] Webserver started at http://127.0.0.1:41729/ using document root <none> and password file <none>
I20260928 21:53:22.785210 23625 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:22.785260 23625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:22.785386 23625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:22.786432 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "057ed1f293e34344b6ccf33c7c065168"
format_stamp: "Formatted at 2026-09-28 21:53:22 on dist-test-slave-jrck"
server_key: "b4d8fdfa3c0e6d9907e617306152533f"
server_key_iv: "44950da6aa4f13d40cc4546ada3191c3"
server_key_version: "encryptionkey@0"
I20260928 21:53:22.786763 23625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "057ed1f293e34344b6ccf33c7c065168"
format_stamp: "Formatted at 2026-09-28 21:53:22 on dist-test-slave-jrck"
server_key: "b4d8fdfa3c0e6d9907e617306152533f"
server_key_iv: "44950da6aa4f13d40cc4546ada3191c3"
server_key_version: "encryptionkey@0"
I20260928 21:53:22.788012 23625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:22.788810 23640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:22.788967 23625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:53:22.789058 23625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "057ed1f293e34344b6ccf33c7c065168"
format_stamp: "Formatted at 2026-09-28 21:53:22 on dist-test-slave-jrck"
server_key: "b4d8fdfa3c0e6d9907e617306152533f"
server_key_iv: "44950da6aa4f13d40cc4546ada3191c3"
server_key_version: "encryptionkey@0"
I20260928 21:53:22.789122 23625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:22.796623 23625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:22.798861 23625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:22.799008 23625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:22.803099 23625 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45215
I20260928 21:53:22.803174 23692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45215 every 8 connection(s)
I20260928 21:53:22.803519 23625 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:22.804590 23693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:22.805595 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 23625
I20260928 21:53:22.805701 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:22.805903 3075 external_mini_cluster.cc:1428] Setting key 9ef2d7d0162447b32dcc3d1a4b787915
I20260928 21:53:22.807310 23693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168: Bootstrap starting.
I20260928 21:53:22.808055 23693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:22.808338 23693 log.cc:824] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:22.809034 23693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168: No bootstrap required, opened a new log
I20260928 21:53:22.810837 23693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057ed1f293e34344b6ccf33c7c065168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260928 21:53:22.810983 23693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:22.811015 23693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057ed1f293e34344b6ccf33c7c065168, State: Initialized, Role: FOLLOWER
I20260928 21:53:22.811144 23693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057ed1f293e34344b6ccf33c7c065168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260928 21:53:22.811221 23693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:22.811262 23693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:22.811311 23693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:22.811877 23693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057ed1f293e34344b6ccf33c7c065168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260928 21:53:22.811993 23693 leader_election.cc:302] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 057ed1f293e34344b6ccf33c7c065168; no voters:
I20260928 21:53:22.812173 23693 leader_election.cc:288] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:22.812207 23698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:22.812377 23698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [term 1 LEADER]: Becoming Leader. State: Replica: 057ed1f293e34344b6ccf33c7c065168, State: Running, Role: LEADER
Sep 28 21:53:22 dist-test-slave-jrck krb5kdc[23612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:53:22.812441 23693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:22.812551 23698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057ed1f293e34344b6ccf33c7c065168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260928 21:53:22.812876 23698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 057ed1f293e34344b6ccf33c7c065168. Latest consensus state: current_term: 1 leader_uuid: "057ed1f293e34344b6ccf33c7c065168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057ed1f293e34344b6ccf33c7c065168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } } }
I20260928 21:53:22.812913 23700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "057ed1f293e34344b6ccf33c7c065168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057ed1f293e34344b6ccf33c7c065168" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } } }
I20260928 21:53:22.813015 23700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:22.813233 23705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:22.813657 23698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:22.813764 23705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:22.815640 23705 catalog_manager.cc:1347] Generated new cluster ID: ed8b68ae8b824c56bbfccd1902d87a9c
I20260928 21:53:22.815690 23705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:22.817057 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:22.806448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53721 (local address 127.0.0.1:45215)
0928 21:53:22.806621 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:22.806631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:22.806656 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:53:22.807079 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:22.807086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:22.807135 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:22.807261 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:22.808054 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:22.809104 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:22.809856 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:22.810264 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:22.810366 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:22.812983 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:22.812988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:22.812988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:22.813000 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:22.815256 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:22.815658 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:22.815662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:22.815662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:22.815790 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:22.816011 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:22.816012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:22.816012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:22.816335 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:22.816520 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:22.816646 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 21:53:22.816812 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260928 21:53:22.818048 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:22.826635 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:22.819593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44785 (local address 127.0.0.1:45215)
0928 21:53:22.819630 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:22.819632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:22.819760 (+ 128us) server_negotiation.cc:407] Connection header received
0928 21:53:22.819793 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:22.819794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:22.819815 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:22.819872 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:22.820498 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:22.821268 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:22.822235 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:22.822560 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:22.822631 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:22.823964 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:22.823966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:22.823966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:22.823973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:22.825553 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:22.825919 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:22.825919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:22.825920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:22.825979 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:22.826138 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:22.826138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:22.826139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:22.826269 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:22.826365 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:22.826455 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 21:53:22.826535 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:53:22.850162 23705 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:22.850962 23705 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:22.861575 23705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 057ed1f293e34344b6ccf33c7c065168: Generated new TSK 0
I20260928 21:53:22.861797 23705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:22.866923 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 23625
2026-09-28T21:53:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:22 dist-test-slave-jrck krb5kdc[23724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:22 dist-test-slave-jrck krb5kdc[23724](info): set up 2 sockets
Sep 28 21:53:22 dist-test-slave-jrck krb5kdc[23724](info): commencing operation
krb5kdc: starting...
W20260928 21:53:23.812345 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:23 dist-test-slave-jrck krb5kdc[23724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:23.923098 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:24.003589 23737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:24.003819 23737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:24.003842 23737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:24.005100 23737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:24.005152 23737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:24.005167 23737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:24.005178 23737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:24.005188 23737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:24.007033 23737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:24.007287 23737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:24.007506 23737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:24.012214 23745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:24.012220 23743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:24.012223 23742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:24.012355 23737 server_base.cc:1034] running on GCE node
I20260928 21:53:24.012722 23737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:24.012993 23737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:24.014143 23737 hybrid_clock.cc:648] HybridClock initialized: now 1790632404014131 us; error 30 us; skew 500 ppm
Sep 28 21:53:24 dist-test-slave-jrck krb5kdc[23724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:24.016557 23737 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:24.016950 23737 webserver.cc:460] Webserver started at http://127.0.0.1:45789/ using document root <none> and password file <none>
I20260928 21:53:24.017125 23737 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:24.017164 23737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:24.017257 23737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:24.018249 23737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "69de23ff10a64369a381069d503161ec"
format_stamp: "Formatted at 2026-09-28 21:53:24 on dist-test-slave-jrck"
server_key: "d56ea3c2a1476f6cf1b9c896f4dec85f"
server_key_iv: "30cff080b02e0a89277b9831b34d54e2"
server_key_version: "encryptionkey@0"
I20260928 21:53:24.018533 23737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "69de23ff10a64369a381069d503161ec"
format_stamp: "Formatted at 2026-09-28 21:53:24 on dist-test-slave-jrck"
server_key: "d56ea3c2a1476f6cf1b9c896f4dec85f"
server_key_iv: "30cff080b02e0a89277b9831b34d54e2"
server_key_version: "encryptionkey@0"
I20260928 21:53:24.019734 23737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:24.020483 23752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:24.020628 23737 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:24.020710 23737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "69de23ff10a64369a381069d503161ec"
format_stamp: "Formatted at 2026-09-28 21:53:24 on dist-test-slave-jrck"
server_key: "d56ea3c2a1476f6cf1b9c896f4dec85f"
server_key_iv: "30cff080b02e0a89277b9831b34d54e2"
server_key_version: "encryptionkey@0"
I20260928 21:53:24.020768 23737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:24.035983 23737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:24.038103 23737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:24.038220 23737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:24.041801 23737 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37831
I20260928 21:53:24.041839 23804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37831 every 8 connection(s)
I20260928 21:53:24.042160 23737 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:24.043098 23805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:24.045369 23805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec: Bootstrap starting.
I20260928 21:53:24.045913 23805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:24.046164 23805 log.cc:824] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:24.046828 23805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec: No bootstrap required, opened a new log
I20260928 21:53:24.047637 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 23737
I20260928 21:53:24.047735 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:24.047906 3075 external_mini_cluster.cc:1428] Setting key ff4489e88b6d4546db93e2bcdef4e275
I20260928 21:53:24.048774 23805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69de23ff10a64369a381069d503161ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } }
I20260928 21:53:24.048949 23805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:24.048995 23805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69de23ff10a64369a381069d503161ec, State: Initialized, Role: FOLLOWER
I20260928 21:53:24.049141 23805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69de23ff10a64369a381069d503161ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } }
I20260928 21:53:24.049227 23805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:24.049254 23805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:24.049304 23805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:24.049990 23805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69de23ff10a64369a381069d503161ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } }
I20260928 21:53:24.050099 23805 leader_election.cc:302] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69de23ff10a64369a381069d503161ec; no voters:
I20260928 21:53:24.050258 23805 leader_election.cc:288] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:24.050367 23810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:24.050493 23805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:24.050509 23810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [term 1 LEADER]: Becoming Leader. State: Replica: 69de23ff10a64369a381069d503161ec, State: Running, Role: LEADER
I20260928 21:53:24.050674 23810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69de23ff10a64369a381069d503161ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } }
I20260928 21:53:24.051043 23810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69de23ff10a64369a381069d503161ec. Latest consensus state: current_term: 1 leader_uuid: "69de23ff10a64369a381069d503161ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69de23ff10a64369a381069d503161ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } } }
I20260928 21:53:24.051189 23812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69de23ff10a64369a381069d503161ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69de23ff10a64369a381069d503161ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37831 } } }
I20260928 21:53:24.051254 23810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:24.051463 23812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:24.051638 23819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:24.052233 23819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:24.054591 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:24.048417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41193 (local address 127.0.0.1:37831)
0928 21:53:24.048658 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:24.048669 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:24.048720 (+ 51us) server_negotiation.cc:407] Connection header received
0928 21:53:24.048837 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:24.048844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:24.048898 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:24.049039 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:24.049297 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:24.050970 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:24.052083 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:24.052630 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:24.052741 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:24.053613 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:24.053617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:24.053618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:24.053935 (+ 317us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:24.053977 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:24.054017 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:24.054099 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:24.054195 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 21:53:24.054291 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":122,"threads_started":1}
I20260928 21:53:24.056945 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:24.057991 23819 catalog_manager.cc:1347] Generated new cluster ID: 853881f1906c4c3082b1379109b5d32d
I20260928 21:53:24.058104 23819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:24.062024 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:24.058372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59829 (local address 127.0.0.1:37831)
0928 21:53:24.058409 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:24.058411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:24.058690 (+ 279us) server_negotiation.cc:407] Connection header received
0928 21:53:24.058786 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:24.058787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:24.058799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:24.058828 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:24.059022 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:24.059703 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:24.060714 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:24.061003 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:24.061044 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:24.061782 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:24.061784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:24.061784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:24.061789 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:24.061804 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:24.061809 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:24.061834 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:24.061924 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 21:53:24.061947 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:53:24.069702 23819 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:24.070438 23819 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:24.076287 23819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69de23ff10a64369a381069d503161ec: Generated new TSK 0
I20260928 21:53:24.076521 23819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:24.085260 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 23737
2026-09-28T21:53:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:24 dist-test-slave-jrck krb5kdc[23836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:24 dist-test-slave-jrck krb5kdc[23836](info): set up 2 sockets
Sep 28 21:53:24 dist-test-slave-jrck krb5kdc[23836](info): commencing operation
krb5kdc: starting...
W20260928 21:53:25.032291 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:25 dist-test-slave-jrck krb5kdc[23836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:25.141043 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:25.221441 23849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:25.221661 23849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:25.221684 23849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:25.222828 23849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:25.222865 23849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:25.222878 23849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:25.222889 23849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:25.222900 23849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:25.224301 23849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:25.224545 23849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:25.224782 23849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:25.229400 23854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:25.229406 23855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:25.229408 23857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:25.229491 23849 server_base.cc:1034] running on GCE node
I20260928 21:53:25.229866 23849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:25.230132 23849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:25.231287 23849 hybrid_clock.cc:648] HybridClock initialized: now 1790632405231271 us; error 29 us; skew 500 ppm
Sep 28 21:53:25 dist-test-slave-jrck krb5kdc[23836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:25.233892 23849 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:25.234336 23849 webserver.cc:460] Webserver started at http://127.0.0.1:35567/ using document root <none> and password file <none>
I20260928 21:53:25.234543 23849 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:25.234632 23849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:25.234764 23849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:25.235833 23849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "a286a10c792940c3981e034b7c04fdc9"
format_stamp: "Formatted at 2026-09-28 21:53:25 on dist-test-slave-jrck"
server_key: "aafde3e0b7404f0aee662e6a2e0d4411"
server_key_iv: "bedc6672e271f9618a9dcfc0f8dfc5f0"
server_key_version: "encryptionkey@0"
I20260928 21:53:25.236167 23849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "a286a10c792940c3981e034b7c04fdc9"
format_stamp: "Formatted at 2026-09-28 21:53:25 on dist-test-slave-jrck"
server_key: "aafde3e0b7404f0aee662e6a2e0d4411"
server_key_iv: "bedc6672e271f9618a9dcfc0f8dfc5f0"
server_key_version: "encryptionkey@0"
I20260928 21:53:25.237459 23849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:25.238193 23864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:25.238349 23849 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:53:25.238431 23849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "a286a10c792940c3981e034b7c04fdc9"
format_stamp: "Formatted at 2026-09-28 21:53:25 on dist-test-slave-jrck"
server_key: "aafde3e0b7404f0aee662e6a2e0d4411"
server_key_iv: "bedc6672e271f9618a9dcfc0f8dfc5f0"
server_key_version: "encryptionkey@0"
I20260928 21:53:25.238490 23849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:25.241797 23849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:25.243536 23849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:25.243661 23849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:25.247527 23849 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44041
I20260928 21:53:25.247562 23916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44041 every 8 connection(s)
I20260928 21:53:25.247855 23849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:25.248798 23917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:25.251008 23917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9: Bootstrap starting.
I20260928 21:53:25.251523 23917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:25.251763 23917 log.cc:824] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:25.252395 23917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9: No bootstrap required, opened a new log
I20260928 21:53:25.253957 23917 raft_consensus.cc:348] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a286a10c792940c3981e034b7c04fdc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260928 21:53:25.254089 23917 raft_consensus.cc:374] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:25.254112 23917 raft_consensus.cc:729] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a286a10c792940c3981e034b7c04fdc9, State: Initialized, Role: FOLLOWER
I20260928 21:53:25.254230 23917 consensus_queue.cc:260] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a286a10c792940c3981e034b7c04fdc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260928 21:53:25.254304 23917 raft_consensus.cc:388] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:25.254326 23917 raft_consensus.cc:482] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:25.254354 23917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:25.254837 23917 raft_consensus.cc:504] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a286a10c792940c3981e034b7c04fdc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260928 21:53:25.254923 23917 leader_election.cc:302] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a286a10c792940c3981e034b7c04fdc9; no voters:
I20260928 21:53:25.255040 23917 leader_election.cc:288] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:25.255090 23920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:25.255264 23917 sys_catalog.cc:564] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:25.255277 23920 raft_consensus.cc:686] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [term 1 LEADER]: Becoming Leader. State: Replica: a286a10c792940c3981e034b7c04fdc9, State: Running, Role: LEADER
I20260928 21:53:25.255394 23920 consensus_queue.cc:237] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a286a10c792940c3981e034b7c04fdc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } }
I20260928 21:53:25.255650 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 23849
I20260928 21:53:25.255764 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:25.255952 3075 external_mini_cluster.cc:1428] Setting key 80d7c9ca9d6a6520c44c044004276e3b
I20260928 21:53:25.256104 23922 sys_catalog.cc:455] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a286a10c792940c3981e034b7c04fdc9. Latest consensus state: current_term: 1 leader_uuid: "a286a10c792940c3981e034b7c04fdc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a286a10c792940c3981e034b7c04fdc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } } }
I20260928 21:53:25.256194 23922 sys_catalog.cc:458] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:25.256335 23921 sys_catalog.cc:455] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a286a10c792940c3981e034b7c04fdc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a286a10c792940c3981e034b7c04fdc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44041 } } }
I20260928 21:53:25.256412 23921 sys_catalog.cc:458] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9 [sys.catalog]: This master's current role is: LEADER
W20260928 21:53:25.257148 23935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:53:25.257241 23935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:53:25.257316 23931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:25.257478 23931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:25.259191 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:25.256504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51491 (local address 127.0.0.1:44041)
0928 21:53:25.256751 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:25.256760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:25.256892 (+ 132us) server_negotiation.cc:407] Connection header received
0928 21:53:25.257021 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:25.257029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:25.257062 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:25.257224 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:25.258702 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:25.258706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:25.258706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:25.258711 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:25.258758 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:25.258773 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:25.258832 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:25.258950 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 21:53:25.258992 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":61,"threads_started":1}
I20260928 21:53:25.259467 23931 catalog_manager.cc:1347] Generated new cluster ID: cab919d88b234f758c3a0eca40815d8d
I20260928 21:53:25.259517 23931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:25.260057 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:25.262627 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:25.261605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37951 (local address 127.0.0.1:44041)
0928 21:53:25.261640 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:25.261641 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:25.261756 (+ 115us) server_negotiation.cc:407] Connection header received
0928 21:53:25.261831 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:25.261832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:25.261837 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:25.261884 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:25.262438 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:25.262439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:25.262439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:25.262443 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:25.262456 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:25.262460 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:25.262506 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:25.262547 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 21:53:25.262559 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:53:25.271384 23931 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:25.271535 23931 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:25.279671 23931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a286a10c792940c3981e034b7c04fdc9: Generated new TSK 0
I20260928 21:53:25.279845 23931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:25.284819 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 23849
2026-09-28T21:53:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1198 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:25 dist-test-slave-jrck krb5kdc[23948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:25 dist-test-slave-jrck krb5kdc[23948](info): set up 2 sockets
Sep 28 21:53:25 dist-test-slave-jrck krb5kdc[23948](info): commencing operation
krb5kdc: starting...
W20260928 21:53:26.232230 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:26 dist-test-slave-jrck krb5kdc[23948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:26.339906 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:26.419071 23961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:26.419287 23961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:26.419309 23961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:26.420413 23961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:26.420475 23961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:26.420498 23961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:26.420513 23961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:26.420529 23961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:26.421993 23961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:26.422197 23961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:26.422367 23961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:26.426962 23967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:26.427057 23961 server_base.cc:1034] running on GCE node
W20260928 21:53:26.426927 23966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:26.427012 23969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:26.427371 23961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:26.427652 23961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:26.428792 23961 hybrid_clock.cc:648] HybridClock initialized: now 1790632406428773 us; error 28 us; skew 500 ppm
Sep 28 21:53:26 dist-test-slave-jrck krb5kdc[23948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:26.434278 23961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:26.434731 23961 webserver.cc:460] Webserver started at http://127.0.0.1:35899/ using document root <none> and password file <none>
I20260928 21:53:26.434916 23961 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:26.434953 23961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:26.435041 23961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:26.436021 23961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "6fd6ac72c07a4b33977bec65dfceafba"
format_stamp: "Formatted at 2026-09-28 21:53:26 on dist-test-slave-jrck"
server_key: "b06768ddc2f36e667e8b7fb499e03418"
server_key_iv: "5550cdea406dfd1fcd4b358c1683936f"
server_key_version: "encryptionkey@0"
I20260928 21:53:26.436314 23961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "6fd6ac72c07a4b33977bec65dfceafba"
format_stamp: "Formatted at 2026-09-28 21:53:26 on dist-test-slave-jrck"
server_key: "b06768ddc2f36e667e8b7fb499e03418"
server_key_iv: "5550cdea406dfd1fcd4b358c1683936f"
server_key_version: "encryptionkey@0"
I20260928 21:53:26.437530 23961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:26.438223 23976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:26.438365 23961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:26.438434 23961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "6fd6ac72c07a4b33977bec65dfceafba"
format_stamp: "Formatted at 2026-09-28 21:53:26 on dist-test-slave-jrck"
server_key: "b06768ddc2f36e667e8b7fb499e03418"
server_key_iv: "5550cdea406dfd1fcd4b358c1683936f"
server_key_version: "encryptionkey@0"
I20260928 21:53:26.438495 23961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:26.461011 23961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:26.463089 23961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:26.463212 23961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:26.466938 23961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45849
I20260928 21:53:26.466974 24028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45849 every 8 connection(s)
I20260928 21:53:26.467299 23961 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:26.468271 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:26.470506 24029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba: Bootstrap starting.
I20260928 21:53:26.471082 24029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:26.471331 24029 log.cc:824] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:26.471978 24029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba: No bootstrap required, opened a new log
I20260928 21:53:26.473531 24029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd6ac72c07a4b33977bec65dfceafba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260928 21:53:26.473690 24029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:26.473713 24029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd6ac72c07a4b33977bec65dfceafba, State: Initialized, Role: FOLLOWER
I20260928 21:53:26.473853 24029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd6ac72c07a4b33977bec65dfceafba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260928 21:53:26.473915 24029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:26.473934 24029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:26.473979 24029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:26.474481 24029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd6ac72c07a4b33977bec65dfceafba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260928 21:53:26.474501 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 23961
I20260928 21:53:26.474575 24029 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd6ac72c07a4b33977bec65dfceafba; no voters:
I20260928 21:53:26.474619 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:26.474740 24029 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:26.474767 24032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:26.474817 3075 external_mini_cluster.cc:1428] Setting key 9a4d42f7e8d9444c54a1559eb3ca1e32
I20260928 21:53:26.474879 24032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [term 1 LEADER]: Becoming Leader. State: Replica: 6fd6ac72c07a4b33977bec65dfceafba, State: Running, Role: LEADER
I20260928 21:53:26.474951 24032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd6ac72c07a4b33977bec65dfceafba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260928 21:53:26.475005 24029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:26.475350 24033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd6ac72c07a4b33977bec65dfceafba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd6ac72c07a4b33977bec65dfceafba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } } }
I20260928 21:53:26.475476 24034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd6ac72c07a4b33977bec65dfceafba. Latest consensus state: current_term: 1 leader_uuid: "6fd6ac72c07a4b33977bec65dfceafba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd6ac72c07a4b33977bec65dfceafba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } } }
I20260928 21:53:26.475488 24033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:26.475575 24034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:26.475822 24038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:26.476368 24038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:26.480546 24038 catalog_manager.cc:1347] Generated new cluster ID: 4e6d8ff7945642cd9677b681cc850b41
I20260928 21:53:26.480594 24038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:53:26 dist-test-slave-jrck krb5kdc[23948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:53:26.486737 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:26.475329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:45377 (local address 127.0.0.1:45849)
0928 21:53:26.475672 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:26.475682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:26.475933 (+ 251us) server_negotiation.cc:407] Connection header received
0928 21:53:26.476156 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:26.476164 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:26.476230 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:26.476398 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:26.477165 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:26.478319 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:26.479358 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:26.479785 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:26.479862 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:26.482564 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:26.482568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:26.482569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:26.482581 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:26.485110 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:26.485518 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:26.485521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:26.485522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:26.485656 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:26.485853 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:26.485853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:26.485854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:26.486162 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:26.486329 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:26.486409 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 21:53:26.486510 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":82,"threads_started":1}
I20260928 21:53:26.487650 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:26.495847 24038 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:26.495977 24037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:26.489110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:49109 (local address 127.0.0.1:45849)
0928 21:53:26.489147 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:26.489149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:26.489223 (+ 74us) server_negotiation.cc:407] Connection header received
0928 21:53:26.489281 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:26.489282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:26.489291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:26.489333 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:26.489921 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:26.490648 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:26.491597 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:26.491891 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:26.491944 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:26.493244 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:26.493246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:26.493246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:26.493253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:26.494711 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:26.495111 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:26.495112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:26.495113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:26.495171 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:26.495373 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:26.495374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:26.495374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:26.495480 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:26.495755 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:26.495851 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 21:53:26.495907 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:53:26.496387 24038 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:26.519837 24038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd6ac72c07a4b33977bec65dfceafba: Generated new TSK 0
I20260928 21:53:26.520040 24038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:26.523073 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 23961
W20260928 21:53:26.526881 23946 connection.cc:535] client connection to 127.0.0.1:45849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45849): Connection reset by peer (error 104)
2026-09-28T21:53:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:26 dist-test-slave-jrck krb5kdc[24060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:26 dist-test-slave-jrck krb5kdc[24060](info): set up 2 sockets
Sep 28 21:53:26 dist-test-slave-jrck krb5kdc[24060](info): commencing operation
krb5kdc: starting...
W20260928 21:53:27.464327 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:27 dist-test-slave-jrck krb5kdc[24060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:27.577392 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:27.656229 24073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:27.656432 24073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:27.656478 24073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:27.657673 24073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:27.657711 24073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:27.657723 24073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:27.657734 24073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:27.657745 24073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:27.659184 24073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:27.659382 24073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:27.659559 24073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:27.664496 24078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:27.664531 24081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:27.664574 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:27.664820 24073 server_base.cc:1034] running on GCE node
I20260928 21:53:27.665294 24073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:27.665699 24073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:27.666888 24073 hybrid_clock.cc:648] HybridClock initialized: now 1790632407666876 us; error 69 us; skew 500 ppm
Sep 28 21:53:27 dist-test-slave-jrck krb5kdc[24060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:27.669401 24073 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:27.669827 24073 webserver.cc:460] Webserver started at http://127.0.0.1:38041/ using document root <none> and password file <none>
I20260928 21:53:27.670043 24073 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:27.670094 24073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:27.670220 24073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:27.671298 24073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "168eef081e344ed781210d5b0db40560"
format_stamp: "Formatted at 2026-09-28 21:53:27 on dist-test-slave-jrck"
server_key: "684d60af0dad1c0b7b295226c0836a92"
server_key_iv: "d87dcc15c0c2379cc6c48436d55bdbbe"
server_key_version: "encryptionkey@0"
I20260928 21:53:27.671633 24073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "168eef081e344ed781210d5b0db40560"
format_stamp: "Formatted at 2026-09-28 21:53:27 on dist-test-slave-jrck"
server_key: "684d60af0dad1c0b7b295226c0836a92"
server_key_iv: "d87dcc15c0c2379cc6c48436d55bdbbe"
server_key_version: "encryptionkey@0"
I20260928 21:53:27.672928 24073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:27.673626 24088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:27.673801 24073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:27.673868 24073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "168eef081e344ed781210d5b0db40560"
format_stamp: "Formatted at 2026-09-28 21:53:27 on dist-test-slave-jrck"
server_key: "684d60af0dad1c0b7b295226c0836a92"
server_key_iv: "d87dcc15c0c2379cc6c48436d55bdbbe"
server_key_version: "encryptionkey@0"
I20260928 21:53:27.673935 24073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:27.710417 24073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:27.712659 24073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:27.712790 24073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:27.716434 24073 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38119
I20260928 21:53:27.716501 24140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38119 every 8 connection(s)
I20260928 21:53:27.716825 24073 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:27.717757 24141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:27.720074 24141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560: Bootstrap starting.
I20260928 21:53:27.720662 24141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:27.720912 24141 log.cc:824] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:27.721551 24141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560: No bootstrap required, opened a new log
I20260928 21:53:27.722114 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 24073
I20260928 21:53:27.722224 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:27.722435 3075 external_mini_cluster.cc:1428] Setting key 42674a85278736215103780ceaa940b8
I20260928 21:53:27.723464 24141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168eef081e344ed781210d5b0db40560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38119 } }
I20260928 21:53:27.723600 24141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:27.723631 24141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 168eef081e344ed781210d5b0db40560, State: Initialized, Role: FOLLOWER
I20260928 21:53:27.723778 24141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168eef081e344ed781210d5b0db40560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38119 } }
I20260928 21:53:27.723862 24141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:27.723892 24141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:27.723937 24141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:27.724638 24141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168eef081e344ed781210d5b0db40560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38119 } }
I20260928 21:53:27.724745 24141 leader_election.cc:302] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 168eef081e344ed781210d5b0db40560; no voters:
I20260928 21:53:27.724891 24141 leader_election.cc:288] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:27.724941 24146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:27.725108 24146 raft_consensus.cc:686] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [term 1 LEADER]: Becoming Leader. State: Replica: 168eef081e344ed781210d5b0db40560, State: Running, Role: LEADER
I20260928 21:53:27.725119 24141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:27.725275 24146 consensus_queue.cc:237] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168eef081e344ed781210d5b0db40560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38119 } }
I20260928 21:53:27.725647 24147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "168eef081e344ed781210d5b0db40560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168eef081e344ed781210d5b0db40560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38119 } } }
I20260928 21:53:27.725639 24146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 168eef081e344ed781210d5b0db40560. Latest consensus state: current_term: 1 leader_uuid: "168eef081e344ed781210d5b0db40560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "168eef081e344ed781210d5b0db40560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38119 } } }
I20260928 21:53:27.725773 24147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:27.725788 24146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:27.725989 24155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:27.726501 24155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:27.728250 24155 catalog_manager.cc:1347] Generated new cluster ID: 7210caba084641bebfc9b97936e53dcf
I20260928 21:53:27.728305 24155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:27.728822 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:27.722919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:52261 (local address 127.0.0.1:38119)
0928 21:53:27.723143 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:27.723151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:27.723177 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:53:27.723424 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:27.723429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:27.723476 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:27.723594 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:27.723820 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:27.725488 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:27.726636 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:27.727232 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:27.727335 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:27.728091 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:27.728096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:27.728097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:27.728314 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:27.728358 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:27.728374 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:27.728474 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:27.728579 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 21:53:27.728609 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":82,"threads_started":1}
I20260928 21:53:27.733435 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:27.737612 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:27.734742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:49757 (local address 127.0.0.1:38119)
0928 21:53:27.734778 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:27.734780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:27.734937 (+ 157us) server_negotiation.cc:407] Connection header received
0928 21:53:27.734995 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:27.734996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:27.735007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:27.735047 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:27.735200 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:27.735683 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:27.736364 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:27.736640 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:27.736697 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:27.737352 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:27.737353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:27.737353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:27.737358 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:27.737373 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:27.737377 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:27.737425 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:27.737493 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 21:53:27.737506 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:53:27.743034 24155 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:27.743551 24155 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:27.747756 24155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 168eef081e344ed781210d5b0db40560: Generated new TSK 0
I20260928 21:53:27.747926 24155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:27.754139 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 24073
W20260928 21:53:27.758178 24058 connection.cc:535] client connection to 127.0.0.1:38119 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38119): Connection reset by peer (error 104)
2026-09-28T21:53:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1231 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:27 dist-test-slave-jrck krb5kdc[24172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:27 dist-test-slave-jrck krb5kdc[24172](info): set up 2 sockets
Sep 28 21:53:27 dist-test-slave-jrck krb5kdc[24172](info): commencing operation
krb5kdc: starting...
W20260928 21:53:28.696534 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:28 dist-test-slave-jrck krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:28.809309 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:28.889405 24185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:28.889619 24185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:28.889642 24185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:28.890797 24185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:28.890844 24185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:28.890856 24185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:28.890868 24185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:28.890879 24185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:28.892385 24185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:28.892634 24185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:28.892850 24185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:28.897521 24193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:28.897560 24190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:28.897522 24191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:28.897775 24185 server_base.cc:1034] running on GCE node
I20260928 21:53:28.898084 24185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:28.898371 24185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:28.899505 24185 hybrid_clock.cc:648] HybridClock initialized: now 1790632408899493 us; error 32 us; skew 500 ppm
Sep 28 21:53:28 dist-test-slave-jrck krb5kdc[24172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:28.902046 24185 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:28.902506 24185 webserver.cc:460] Webserver started at http://127.0.0.1:37007/ using document root <none> and password file <none>
I20260928 21:53:28.902763 24185 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:28.902813 24185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:28.902940 24185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:28.904011 24185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "4a14fcc159934ad1aa7a6085fe69f947"
format_stamp: "Formatted at 2026-09-28 21:53:28 on dist-test-slave-jrck"
server_key: "183a5fd9ab9682b5a318c96cf2a3d1a7"
server_key_iv: "8878ea8373a7a781c4e83069c8cd75ac"
server_key_version: "encryptionkey@0"
I20260928 21:53:28.904346 24185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "4a14fcc159934ad1aa7a6085fe69f947"
format_stamp: "Formatted at 2026-09-28 21:53:28 on dist-test-slave-jrck"
server_key: "183a5fd9ab9682b5a318c96cf2a3d1a7"
server_key_iv: "8878ea8373a7a781c4e83069c8cd75ac"
server_key_version: "encryptionkey@0"
I20260928 21:53:28.905732 24185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:53:28.906472 24200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:28.906654 24185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:53:28.906741 24185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "4a14fcc159934ad1aa7a6085fe69f947"
format_stamp: "Formatted at 2026-09-28 21:53:28 on dist-test-slave-jrck"
server_key: "183a5fd9ab9682b5a318c96cf2a3d1a7"
server_key_iv: "8878ea8373a7a781c4e83069c8cd75ac"
server_key_version: "encryptionkey@0"
I20260928 21:53:28.906808 24185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:28.910178 24185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:28.912237 24185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:28.912369 24185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:28.916121 24185 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33243
I20260928 21:53:28.916155 24252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33243 every 8 connection(s)
I20260928 21:53:28.916491 24185 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:28.917457 24253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:28.919785 24253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947: Bootstrap starting.
I20260928 21:53:28.920329 24253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:28.920639 24253 log.cc:824] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:28.921331 24253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947: No bootstrap required, opened a new log
I20260928 21:53:28.922951 24253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a14fcc159934ad1aa7a6085fe69f947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } }
I20260928 21:53:28.923099 24253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:28.923123 24253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a14fcc159934ad1aa7a6085fe69f947, State: Initialized, Role: FOLLOWER
I20260928 21:53:28.923252 24253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a14fcc159934ad1aa7a6085fe69f947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } }
I20260928 21:53:28.923312 24253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:28.923334 24253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:28.923378 24253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:28.923831 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 24185
I20260928 21:53:28.923930 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:28.923939 24253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a14fcc159934ad1aa7a6085fe69f947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } }
I20260928 21:53:28.924048 24253 leader_election.cc:302] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a14fcc159934ad1aa7a6085fe69f947; no voters:
I20260928 21:53:28.924124 3075 external_mini_cluster.cc:1428] Setting key 321075f381bca89f8932e346d889fb8d
I20260928 21:53:28.924202 24253 leader_election.cc:288] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:28.924227 24256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:28.924335 24256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [term 1 LEADER]: Becoming Leader. State: Replica: 4a14fcc159934ad1aa7a6085fe69f947, State: Running, Role: LEADER
I20260928 21:53:28.924500 24253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:28.924503 24256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a14fcc159934ad1aa7a6085fe69f947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } }
I20260928 21:53:28.924975 24257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a14fcc159934ad1aa7a6085fe69f947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a14fcc159934ad1aa7a6085fe69f947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } } }
I20260928 21:53:28.924971 24258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a14fcc159934ad1aa7a6085fe69f947. Latest consensus state: current_term: 1 leader_uuid: "4a14fcc159934ad1aa7a6085fe69f947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a14fcc159934ad1aa7a6085fe69f947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33243 } } }
I20260928 21:53:28.925137 24258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:28.925097 24257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:28.925441 24263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:28.925971 24263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:28.927249 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:28.924775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:52461 (local address 127.0.0.1:33243)
0928 21:53:28.924937 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:28.924947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:28.925247 (+ 300us) server_negotiation.cc:407] Connection header received
0928 21:53:28.925374 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:28.925381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:28.925416 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:28.925550 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:28.926797 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:28.926800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:28.926801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:28.926806 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:28.926850 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:28.926865 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:28.926946 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:28.927015 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 21:53:28.927062 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260928 21:53:28.927949 24263 catalog_manager.cc:1347] Generated new cluster ID: 4e322f7d071f4c5fbbadd6c3325aaadb
I20260928 21:53:28.927999 24263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:28.928193 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:28.930596 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:28.929553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:51693 (local address 127.0.0.1:33243)
0928 21:53:28.929587 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:28.929589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:28.929708 (+ 119us) server_negotiation.cc:407] Connection header received
0928 21:53:28.929765 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:28.929766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:28.929772 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:28.929823 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:28.930379 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:28.930380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:28.930381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:28.930385 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:28.930400 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:28.930404 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:28.930452 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:28.930509 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:53:28.930530 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:53:28.957105 24263 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:28.957330 24263 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:28.965935 24263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a14fcc159934ad1aa7a6085fe69f947: Generated new TSK 0
I20260928 21:53:28.966208 24263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:28.978859 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 24185
2026-09-28T21:53:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:29 dist-test-slave-jrck krb5kdc[24284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:29 dist-test-slave-jrck krb5kdc[24284](info): set up 2 sockets
Sep 28 21:53:29 dist-test-slave-jrck krb5kdc[24284](info): commencing operation
krb5kdc: starting...
W20260928 21:53:29.924256 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:29 dist-test-slave-jrck krb5kdc[24284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:30.035321 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:30.117588 24297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:30.117817 24297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:30.117842 24297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:30.119033 24297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:30.119069 24297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:30.119083 24297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:30.119094 24297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:30.119104 24297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:30.120662 24297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:30.120868 24297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:30.121065 24297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:30.126242 24302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:30.126312 24303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:30.126271 24305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:30.126370 24297 server_base.cc:1034] running on GCE node
I20260928 21:53:30.126686 24297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:30.127020 24297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:30.128166 24297 hybrid_clock.cc:648] HybridClock initialized: now 1790632410128144 us; error 59 us; skew 500 ppm
Sep 28 21:53:30 dist-test-slave-jrck krb5kdc[24284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:30.130793 24297 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:30.131232 24297 webserver.cc:460] Webserver started at http://127.0.0.1:35495/ using document root <none> and password file <none>
I20260928 21:53:30.131454 24297 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:30.131503 24297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:30.131629 24297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:30.132761 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "5cadf2c60f6e466a8d8228c8302887d0"
format_stamp: "Formatted at 2026-09-28 21:53:30 on dist-test-slave-jrck"
server_key: "d46ff0977d075de68bdca163d18e18e5"
server_key_iv: "736d999a0d8c03666d97829161357abf"
server_key_version: "encryptionkey@0"
I20260928 21:53:30.133090 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "5cadf2c60f6e466a8d8228c8302887d0"
format_stamp: "Formatted at 2026-09-28 21:53:30 on dist-test-slave-jrck"
server_key: "d46ff0977d075de68bdca163d18e18e5"
server_key_iv: "736d999a0d8c03666d97829161357abf"
server_key_version: "encryptionkey@0"
I20260928 21:53:30.134320 24297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:30.135004 24312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:30.135154 24297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:30.135245 24297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "5cadf2c60f6e466a8d8228c8302887d0"
format_stamp: "Formatted at 2026-09-28 21:53:30 on dist-test-slave-jrck"
server_key: "d46ff0977d075de68bdca163d18e18e5"
server_key_iv: "736d999a0d8c03666d97829161357abf"
server_key_version: "encryptionkey@0"
I20260928 21:53:30.135309 24297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:30.153671 24297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:30.155927 24297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:30.156075 24297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:30.159991 24297 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32887
I20260928 21:53:30.160040 24364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32887 every 8 connection(s)
I20260928 21:53:30.160390 24297 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:30.161367 24365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:30.163734 24365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0: Bootstrap starting.
I20260928 21:53:30.164297 24365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:30.164609 24365 log.cc:824] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:30.165350 24365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0: No bootstrap required, opened a new log
I20260928 21:53:30.167017 24365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cadf2c60f6e466a8d8228c8302887d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32887 } }
I20260928 21:53:30.167173 24365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:30.167194 24365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cadf2c60f6e466a8d8228c8302887d0, State: Initialized, Role: FOLLOWER
I20260928 21:53:30.167315 24365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cadf2c60f6e466a8d8228c8302887d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32887 } }
I20260928 21:53:30.167366 24365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:30.167384 24365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:30.167415 24365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:30.167953 24365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cadf2c60f6e466a8d8228c8302887d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32887 } }
I20260928 21:53:30.168041 24365 leader_election.cc:302] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cadf2c60f6e466a8d8228c8302887d0; no voters:
I20260928 21:53:30.168177 24365 leader_election.cc:288] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:30.168223 24368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:30.168357 24368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [term 1 LEADER]: Becoming Leader. State: Replica: 5cadf2c60f6e466a8d8228c8302887d0, State: Running, Role: LEADER
I20260928 21:53:30.168475 24368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cadf2c60f6e466a8d8228c8302887d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32887 } }
I20260928 21:53:30.168494 24365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:30.168862 24370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cadf2c60f6e466a8d8228c8302887d0. Latest consensus state: current_term: 1 leader_uuid: "5cadf2c60f6e466a8d8228c8302887d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cadf2c60f6e466a8d8228c8302887d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32887 } } }
I20260928 21:53:30.168998 24369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cadf2c60f6e466a8d8228c8302887d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cadf2c60f6e466a8d8228c8302887d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32887 } } }
I20260928 21:53:30.169001 24370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:30.169092 24369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:30.169570 24377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:30.170079 24377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:30.170169 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 24297
I20260928 21:53:30.170284 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:30.170464 3075 external_mini_cluster.cc:1428] Setting key fe45dabd572d77cca1f68b49fba432cf
I20260928 21:53:30.171787 24377 catalog_manager.cc:1347] Generated new cluster ID: a40879d779494f0283176d31d391b1d0
I20260928 21:53:30.171840 24377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:53:30 dist-test-slave-jrck krb5kdc[24284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:53:30.182864 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:30.170910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:57851 (local address 127.0.0.1:32887)
0928 21:53:30.171346 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:30.171356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:30.171387 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:53:30.171510 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:30.171518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:30.171580 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:30.172560 (+ 980us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:30.172668 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:30.174164 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:30.175335 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:30.176026 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:30.176159 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:30.179077 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:30.179083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:30.179083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:30.179098 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:30.181079 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:30.181555 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:30.181558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:30.181559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:30.181681 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:30.181900 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:30.181901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:30.181901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:30.182246 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:30.182379 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:30.182519 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 21:53:30.182622 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":67,"threads_started":1}
I20260928 21:53:30.183820 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:30.189610 24377 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:30.190451 24377 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:30.197293 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:30.189376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:56167 (local address 127.0.0.1:32887)
0928 21:53:30.189428 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:30.189430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:30.189657 (+ 227us) server_negotiation.cc:407] Connection header received
0928 21:53:30.189765 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:30.189766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:30.189779 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:30.189849 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:30.190516 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:30.191592 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:30.192619 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:30.192960 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:30.193032 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:30.194443 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:30.194445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:30.194445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:30.194454 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:30.196014 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:30.196445 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:30.196446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:30.196447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:30.196549 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:30.196747 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:30.196747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:30.196748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:30.196878 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:30.196979 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:30.197126 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 21:53:30.197192 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:53:30.199977 24377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cadf2c60f6e466a8d8228c8302887d0: Generated new TSK 0
I20260928 21:53:30.200187 24377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:30.202867 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 24297
W20260928 21:53:30.206959 24282 connection.cc:535] client connection to 127.0.0.1:32887 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32887): Connection reset by peer (error 104)
2026-09-28T21:53:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:30 dist-test-slave-jrck krb5kdc[24396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:30 dist-test-slave-jrck krb5kdc[24396](info): set up 2 sockets
Sep 28 21:53:30 dist-test-slave-jrck krb5kdc[24396](info): commencing operation
krb5kdc: starting...
W20260928 21:53:31.156308 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:31 dist-test-slave-jrck krb5kdc[24396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:31.268952 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:31.348835 24409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:31.349058 24409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:31.349099 24409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:31.350291 24409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:31.350337 24409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:31.350355 24409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:31.350373 24409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:31.350386 24409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:31.351904 24409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:31.352172 24409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:31.352387 24409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:31.357080 24414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:31.357088 24415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:31.357087 24417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:31.357585 24409 server_base.cc:1034] running on GCE node
I20260928 21:53:31.357822 24409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:31.358098 24409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:31.359237 24409 hybrid_clock.cc:648] HybridClock initialized: now 1790632411359221 us; error 34 us; skew 500 ppm
Sep 28 21:53:31 dist-test-slave-jrck krb5kdc[24396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:31.361750 24409 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:31.362190 24409 webserver.cc:460] Webserver started at http://127.0.0.1:33053/ using document root <none> and password file <none>
I20260928 21:53:31.362406 24409 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:31.362452 24409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:31.362574 24409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:31.363641 24409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "05ab07277b284afb91e2e20bb60992d3"
format_stamp: "Formatted at 2026-09-28 21:53:31 on dist-test-slave-jrck"
server_key: "e2bdcf34dd6acd9c0f2e6b8b4841cce2"
server_key_iv: "b09ece6bcbef1bdba4098210e2606213"
server_key_version: "encryptionkey@0"
I20260928 21:53:31.363965 24409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "05ab07277b284afb91e2e20bb60992d3"
format_stamp: "Formatted at 2026-09-28 21:53:31 on dist-test-slave-jrck"
server_key: "e2bdcf34dd6acd9c0f2e6b8b4841cce2"
server_key_iv: "b09ece6bcbef1bdba4098210e2606213"
server_key_version: "encryptionkey@0"
I20260928 21:53:31.365310 24409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:31.366047 24424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:31.366232 24409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:53:31.366330 24409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "05ab07277b284afb91e2e20bb60992d3"
format_stamp: "Formatted at 2026-09-28 21:53:31 on dist-test-slave-jrck"
server_key: "e2bdcf34dd6acd9c0f2e6b8b4841cce2"
server_key_iv: "b09ece6bcbef1bdba4098210e2606213"
server_key_version: "encryptionkey@0"
I20260928 21:53:31.366389 24409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:31.376621 24409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:31.378818 24409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:31.378939 24409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:31.382680 24409 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45199
I20260928 21:53:31.382721 24476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45199 every 8 connection(s)
I20260928 21:53:31.383044 24409 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:31.383432 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 24409
I20260928 21:53:31.383566 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:31.383777 3075 external_mini_cluster.cc:1428] Setting key c897e51ef740e7b6250441a1626be6c8
I20260928 21:53:31.384130 24477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:31.386982 24477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3: Bootstrap starting.
I20260928 21:53:31.387537 24477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:31.387806 24477 log.cc:824] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:31.388425 24477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3: No bootstrap required, opened a new log
I20260928 21:53:31.389267 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:31.384300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:41357 (local address 127.0.0.1:45199)
0928 21:53:31.384523 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:31.384534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:31.384564 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:53:31.385274 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:31.385281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:31.385337 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:31.385469 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:31.385701 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:31.386609 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:31.387429 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:31.387895 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:31.388002 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:31.388688 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:31.388693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:31.388693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:31.388854 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:31.388886 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:31.388896 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:31.388963 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:31.389057 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 21:53:31.389094 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260928 21:53:31.390200 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:31.390604 24477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ab07277b284afb91e2e20bb60992d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } }
I20260928 21:53:31.390743 24477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:31.390774 24477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ab07277b284afb91e2e20bb60992d3, State: Initialized, Role: FOLLOWER
I20260928 21:53:31.390890 24477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ab07277b284afb91e2e20bb60992d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } }
I20260928 21:53:31.390962 24477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:31.390991 24477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:31.391036 24477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:31.391762 24477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ab07277b284afb91e2e20bb60992d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } }
I20260928 21:53:31.391865 24477 leader_election.cc:302] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05ab07277b284afb91e2e20bb60992d3; no voters:
I20260928 21:53:31.392022 24477 leader_election.cc:288] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:31.392063 24487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:31.392226 24487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [term 1 LEADER]: Becoming Leader. State: Replica: 05ab07277b284afb91e2e20bb60992d3, State: Running, Role: LEADER
I20260928 21:53:31.392278 24477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:31.392319 24487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ab07277b284afb91e2e20bb60992d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } }
I20260928 21:53:31.392771 24489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ab07277b284afb91e2e20bb60992d3. Latest consensus state: current_term: 1 leader_uuid: "05ab07277b284afb91e2e20bb60992d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ab07277b284afb91e2e20bb60992d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } } }
I20260928 21:53:31.392905 24489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:31.393108 24488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05ab07277b284afb91e2e20bb60992d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ab07277b284afb91e2e20bb60992d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45199 } } }
I20260928 21:53:31.393219 24488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:31.393466 24496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:31.394012 24496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:31.395110 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:31.391572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:42727 (local address 127.0.0.1:45199)
0928 21:53:31.391617 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:31.391618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:31.391703 (+ 85us) server_negotiation.cc:407] Connection header received
0928 21:53:31.391765 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:31.391766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:31.391775 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:31.391819 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:31.391959 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:31.392646 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:31.393674 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:31.393963 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:31.394330 (+ 367us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:31.394801 (+ 471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:31.394802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:31.394802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:31.394807 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:31.394822 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:31.394826 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:31.394994 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:31.395018 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 21:53:31.395028 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:53:31.395632 24496 catalog_manager.cc:1347] Generated new cluster ID: 27f32aca473c47d7b730ec312cafb7d8
I20260928 21:53:31.395677 24496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:31.408694 24496 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:31.409233 24496 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:31.414741 24496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05ab07277b284afb91e2e20bb60992d3: Generated new TSK 0
I20260928 21:53:31.414933 24496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:31.417903 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 24409
W20260928 21:53:31.421617 24394 connection.cc:535] client connection to 127.0.0.1:45199 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45199): Connection reset by peer (error 104)
2026-09-28T21:53:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:31 dist-test-slave-jrck krb5kdc[24508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:31 dist-test-slave-jrck krb5kdc[24508](info): set up 2 sockets
Sep 28 21:53:31 dist-test-slave-jrck krb5kdc[24508](info): commencing operation
krb5kdc: starting...
W20260928 21:53:32.377897 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:32 dist-test-slave-jrck krb5kdc[24508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:53:32.496055 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:32.578540 24521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:32.578758 24521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:32.578781 24521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:32.580005 24521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:32.580052 24521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:32.580065 24521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:32.580076 24521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:32.580087 24521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:32.581689 24521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:32.581925 24521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:32.582139 24521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:32.586632 24526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:32.586752 24521 server_base.cc:1034] running on GCE node
W20260928 21:53:32.586649 24529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:32.586627 24527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:32.587047 24521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:32.587330 24521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:32.588488 24521 hybrid_clock.cc:648] HybridClock initialized: now 1790632412588473 us; error 28 us; skew 500 ppm
Sep 28 21:53:32 dist-test-slave-jrck krb5kdc[24508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:32.590878 24521 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:53:32.591269 24521 webserver.cc:460] Webserver started at http://127.0.0.1:35043/ using document root <none> and password file <none>
I20260928 21:53:32.591470 24521 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:32.591518 24521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:32.591639 24521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:32.592749 24521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "a43bc34ce24444cc96c75f1a9f423fb7"
format_stamp: "Formatted at 2026-09-28 21:53:32 on dist-test-slave-jrck"
server_key: "4e11506add6ca11d7019dcf8b3377bcb"
server_key_iv: "efc8aa6b41de41924acb4da3ec6e0cda"
server_key_version: "encryptionkey@0"
I20260928 21:53:32.593070 24521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "a43bc34ce24444cc96c75f1a9f423fb7"
format_stamp: "Formatted at 2026-09-28 21:53:32 on dist-test-slave-jrck"
server_key: "4e11506add6ca11d7019dcf8b3377bcb"
server_key_iv: "efc8aa6b41de41924acb4da3ec6e0cda"
server_key_version: "encryptionkey@0"
I20260928 21:53:32.594290 24521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:53:32.595005 24536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:32.595165 24521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:32.595245 24521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "a43bc34ce24444cc96c75f1a9f423fb7"
format_stamp: "Formatted at 2026-09-28 21:53:32 on dist-test-slave-jrck"
server_key: "4e11506add6ca11d7019dcf8b3377bcb"
server_key_iv: "efc8aa6b41de41924acb4da3ec6e0cda"
server_key_version: "encryptionkey@0"
I20260928 21:53:32.595304 24521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:32.598637 24521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:32.600770 24521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:32.600901 24521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:32.604616 24521 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43343
I20260928 21:53:32.604697 24588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43343 every 8 connection(s)
I20260928 21:53:32.604991 24521 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:32.605839 24589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:32.608104 24589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7: Bootstrap starting.
I20260928 21:53:32.608708 24589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:32.608961 24589 log.cc:824] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:32.609673 24589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7: No bootstrap required, opened a new log
I20260928 21:53:32.610903 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 24521
I20260928 21:53:32.611011 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:32.611167 3075 external_mini_cluster.cc:1428] Setting key 643b7a40f7468b375a33f6d2991d51e1
I20260928 21:53:32.611320 24589 raft_consensus.cc:348] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43bc34ce24444cc96c75f1a9f423fb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43343 } }
I20260928 21:53:32.611465 24589 raft_consensus.cc:374] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:32.611497 24589 raft_consensus.cc:729] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43bc34ce24444cc96c75f1a9f423fb7, State: Initialized, Role: FOLLOWER
I20260928 21:53:32.611886 24589 consensus_queue.cc:260] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43bc34ce24444cc96c75f1a9f423fb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43343 } }
I20260928 21:53:32.611967 24589 raft_consensus.cc:388] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:32.611999 24589 raft_consensus.cc:482] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:32.612046 24589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:32.612950 24589 raft_consensus.cc:504] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43bc34ce24444cc96c75f1a9f423fb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43343 } }
I20260928 21:53:32.613050 24589 leader_election.cc:302] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a43bc34ce24444cc96c75f1a9f423fb7; no voters:
I20260928 21:53:32.613214 24589 leader_election.cc:288] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:32.613245 24594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:32.613421 24594 raft_consensus.cc:686] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [term 1 LEADER]: Becoming Leader. State: Replica: a43bc34ce24444cc96c75f1a9f423fb7, State: Running, Role: LEADER
I20260928 21:53:32.613495 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:32.611657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:56201 (local address 127.0.0.1:43343)
0928 21:53:32.611834 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:32.611842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:32.611875 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:53:32.611984 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:32.611989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:32.612019 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:32.612157 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:32.613048 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:32.613052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:32.613052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:32.613057 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:32.613105 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:32.613121 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:32.613182 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:32.613244 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:53:32.613291 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260928 21:53:32.613540 24594 consensus_queue.cc:237] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43bc34ce24444cc96c75f1a9f423fb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43343 } }
I20260928 21:53:32.613533 24589 sys_catalog.cc:564] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:32.613878 24594 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a43bc34ce24444cc96c75f1a9f423fb7. Latest consensus state: current_term: 1 leader_uuid: "a43bc34ce24444cc96c75f1a9f423fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43bc34ce24444cc96c75f1a9f423fb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43343 } } }
I20260928 21:53:32.614017 24594 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:32.613888 24595 sys_catalog.cc:455] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a43bc34ce24444cc96c75f1a9f423fb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a43bc34ce24444cc96c75f1a9f423fb7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43343 } } }
I20260928 21:53:32.614231 24595 sys_catalog.cc:458] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:32.614393 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:32.614550 24598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:32.615054 24598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:32.616846 24598 catalog_manager.cc:1347] Generated new cluster ID: 12ea102e043b48ae9e1b20b09d2472fb
I20260928 21:53:32.616946 24598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:32.617166 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:32.615989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:34977 (local address 127.0.0.1:43343)
0928 21:53:32.616030 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:32.616031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:32.616176 (+ 145us) server_negotiation.cc:407] Connection header received
0928 21:53:32.616263 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:32.616264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:32.616269 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:32.616315 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:32.616919 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:32.616920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:32.616920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:32.616925 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:32.616941 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:32.616945 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:32.617004 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:32.617095 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:53:32.617109 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 21:53:32.623505 24598 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:32.623675 24598 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:32.636721 24598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a43bc34ce24444cc96c75f1a9f423fb7: Generated new TSK 0
I20260928 21:53:32.636929 24598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:32.641115 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 24521
2026-09-28T21:53:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:32 dist-test-slave-jrck krb5kdc[24620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:32 dist-test-slave-jrck krb5kdc[24620](info): set up 2 sockets
Sep 28 21:53:32 dist-test-slave-jrck krb5kdc[24620](info): commencing operation
krb5kdc: starting...
W20260928 21:53:33.580616 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:33 dist-test-slave-jrck krb5kdc[24620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:33.705533 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33633
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:43949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:33633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:33.785336 24633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:33.785576 24633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:33.785611 24633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:33.786816 24633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:33.786865 24633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:33.786883 24633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:33.786899 24633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:33.786914 24633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:33.788514 24633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:43949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:33633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:33.788786 24633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:33.789002 24633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:33.793802 24638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:33.793763 24639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:33.793763 24641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:33.794103 24633 server_base.cc:1034] running on GCE node
I20260928 21:53:33.794330 24633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:33.794633 24633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:33.795783 24633 hybrid_clock.cc:648] HybridClock initialized: now 1790632413795769 us; error 32 us; skew 500 ppm
Sep 28 21:53:33 dist-test-slave-jrck krb5kdc[24620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632413, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:33.798363 24633 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:33.798834 24633 webserver.cc:460] Webserver started at http://127.3.0.254:40033/ using document root <none> and password file <none>
I20260928 21:53:33.799082 24633 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:33.799134 24633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:33.799260 24633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:33.800387 24633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "9ba79f24cf6442d9a924532f1a9ffd4e"
format_stamp: "Formatted at 2026-09-28 21:53:33 on dist-test-slave-jrck"
server_key: "35b2c76a8fb0c3d0bb5530c99987d883"
server_key_iv: "d3c5911bcb22f14118e636ddc10240da"
server_key_version: "encryptionkey@0"
I20260928 21:53:33.800773 24633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "9ba79f24cf6442d9a924532f1a9ffd4e"
format_stamp: "Formatted at 2026-09-28 21:53:33 on dist-test-slave-jrck"
server_key: "35b2c76a8fb0c3d0bb5530c99987d883"
server_key_iv: "d3c5911bcb22f14118e636ddc10240da"
server_key_version: "encryptionkey@0"
I20260928 21:53:33.802042 24633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:33.802742 24648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:33.802910 24633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:33.803011 24633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "9ba79f24cf6442d9a924532f1a9ffd4e"
format_stamp: "Formatted at 2026-09-28 21:53:33 on dist-test-slave-jrck"
server_key: "35b2c76a8fb0c3d0bb5530c99987d883"
server_key_iv: "d3c5911bcb22f14118e636ddc10240da"
server_key_version: "encryptionkey@0"
I20260928 21:53:33.803081 24633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:33.833387 24633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:33.835649 24633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:33.835793 24633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:33.839546 24633 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:33633
I20260928 21:53:33.839583 24700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:33633 every 8 connection(s)
I20260928 21:53:33.839917 24633 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:33.840608 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 24633
I20260928 21:53:33.840713 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:33.840850 24701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:33.840902 3075 external_mini_cluster.cc:1428] Setting key 1f98ed40a59ae9fa917f1ae3b3adf2a9
I20260928 21:53:33.843976 24701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e: Bootstrap starting.
I20260928 21:53:33.844681 24701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:33.844944 24701 log.cc:824] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:33.845608 24701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e: No bootstrap required, opened a new log
Sep 28 21:53:33 dist-test-slave-jrck krb5kdc[24620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:33.847994 24701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba79f24cf6442d9a924532f1a9ffd4e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33633 } }
I20260928 21:53:33.848147 24701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:33.848181 24701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba79f24cf6442d9a924532f1a9ffd4e, State: Initialized, Role: FOLLOWER
I20260928 21:53:33.848306 24701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba79f24cf6442d9a924532f1a9ffd4e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33633 } }
I20260928 21:53:33.848389 24701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:33.848420 24701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:33.848498 24701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:33.849222 24701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba79f24cf6442d9a924532f1a9ffd4e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33633 } }
I20260928 21:53:33.849342 24701 leader_election.cc:302] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ba79f24cf6442d9a924532f1a9ffd4e; no voters:
I20260928 21:53:33.849526 24701 leader_election.cc:288] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:33.849560 24706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:33.849694 24706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [term 1 LEADER]: Becoming Leader. State: Replica: 9ba79f24cf6442d9a924532f1a9ffd4e, State: Running, Role: LEADER
I20260928 21:53:33.849773 24701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:33.849819 24706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba79f24cf6442d9a924532f1a9ffd4e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33633 } }
I20260928 21:53:33.850215 24707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ba79f24cf6442d9a924532f1a9ffd4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba79f24cf6442d9a924532f1a9ffd4e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33633 } } }
I20260928 21:53:33.850307 24707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:33.850497 24713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:33.850219 24708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ba79f24cf6442d9a924532f1a9ffd4e. Latest consensus state: current_term: 1 leader_uuid: "9ba79f24cf6442d9a924532f1a9ffd4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba79f24cf6442d9a924532f1a9ffd4e" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 33633 } } }
I20260928 21:53:33.850591 24708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:33.851018 24713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:33.852068 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:33.841480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47479 (local address 127.3.0.254:33633)
0928 21:53:33.841688 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:33.841698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:33.841734 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:53:33.842249 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:33.842260 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:33.842328 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:33.842483 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:33.843067 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:33.844489 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:33.845283 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:33.845753 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:33.845842 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:33.848369 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:33.848373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:33.848373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:33.848386 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:33.850202 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:33.850686 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:33.850690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:33.850690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:33.850800 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:33.851052 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:33.851053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:33.851053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:33.851375 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:33.851519 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:33.851763 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 21:53:33.851860 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260928 21:53:33.852720 24713 catalog_manager.cc:1347] Generated new cluster ID: b2ca366d0844492193e98225b639b2a0
I20260928 21:53:33.852766 24713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:33.853051 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:33.859831 24704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:33.854322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50687 (local address 127.3.0.254:33633)
0928 21:53:33.854356 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:33.854357 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:33.854467 (+ 110us) server_negotiation.cc:407] Connection header received
0928 21:53:33.854529 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:33.854530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:33.854540 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:33.854584 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:33.855032 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:33.855507 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:33.856154 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:33.856384 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:33.856436 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:33.857498 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:33.857500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:33.857500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:33.857507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:33.858768 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:33.859095 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:33.859096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:33.859096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:33.859146 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:33.859367 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:33.859367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:33.859368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:33.859474 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:33.859584 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:33.859693 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 21:53:33.859749 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:53:33.874550 24713 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:33.875094 24713 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:33.878031 24713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ba79f24cf6442d9a924532f1a9ffd4e: Generated new TSK 0
I20260928 21:53:33.878252 24713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:33.888203 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 24633
W20260928 21:53:33.892035 24618 connection.cc:535] client connection to 127.3.0.254:33633 recv error: Network error: failed to read from TLS socket (remote: 127.3.0.254:33633): Connection reset by peer (error 104)
2026-09-28T21:53:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1246 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:33 dist-test-slave-jrck krb5kdc[24732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:33 dist-test-slave-jrck krb5kdc[24732](info): set up 2 sockets
Sep 28 21:53:33 dist-test-slave-jrck krb5kdc[24732](info): commencing operation
krb5kdc: starting...
W20260928 21:53:34.836395 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:34 dist-test-slave-jrck krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:34.952054 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:43303
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:43303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:35.031064 24745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:35.031286 24745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:35.031318 24745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:35.032493 24745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:35.032534 24745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:35.032552 24745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:35.032568 24745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:35.032583 24745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:35.034075 24745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:43303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:43303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:35.034336 24745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:35.034580 24745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:35.040930 24753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:35.040933 24751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:35.041186 24750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:35.041040 24745 server_base.cc:1034] running on GCE node
I20260928 21:53:35.041513 24745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:35.041815 24745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:35.042959 24745 hybrid_clock.cc:648] HybridClock initialized: now 1790632415042947 us; error 32 us; skew 500 ppm
Sep 28 21:53:35 dist-test-slave-jrck krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632415, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:35.045405 24745 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:35.045811 24745 webserver.cc:460] Webserver started at http://127.3.0.254:44283/ using document root <none> and password file <none>
I20260928 21:53:35.045986 24745 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:35.046025 24745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:35.046141 24745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:35.047173 24745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "77eb1f982e7c407eba9c019ad9406719"
format_stamp: "Formatted at 2026-09-28 21:53:35 on dist-test-slave-jrck"
server_key: "71c3c979f5967fc449c10b1e022dbff4"
server_key_iv: "1a5e7aa615516dbe34f1f0c9eb65755c"
server_key_version: "encryptionkey@0"
I20260928 21:53:35.047458 24745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "77eb1f982e7c407eba9c019ad9406719"
format_stamp: "Formatted at 2026-09-28 21:53:35 on dist-test-slave-jrck"
server_key: "71c3c979f5967fc449c10b1e022dbff4"
server_key_iv: "1a5e7aa615516dbe34f1f0c9eb65755c"
server_key_version: "encryptionkey@0"
I20260928 21:53:35.048709 24745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 21:53:35.049358 24760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:35.049520 24745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:35.049611 24745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "77eb1f982e7c407eba9c019ad9406719"
format_stamp: "Formatted at 2026-09-28 21:53:35 on dist-test-slave-jrck"
server_key: "71c3c979f5967fc449c10b1e022dbff4"
server_key_iv: "1a5e7aa615516dbe34f1f0c9eb65755c"
server_key_version: "encryptionkey@0"
I20260928 21:53:35.049670 24745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:35.063550 24745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:35.065395 24745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:35.065505 24745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:35.069154 24745 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:43303
I20260928 21:53:35.069187 24812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:43303 every 8 connection(s)
I20260928 21:53:35.069499 24745 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:35.070330 24813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:35.072571 24813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719: Bootstrap starting.
I20260928 21:53:35.073167 24813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:35.073421 24813 log.cc:824] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:35.074043 24813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719: No bootstrap required, opened a new log
I20260928 21:53:35.075635 24813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77eb1f982e7c407eba9c019ad9406719" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43303 } }
I20260928 21:53:35.075768 24813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:35.075799 24813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77eb1f982e7c407eba9c019ad9406719, State: Initialized, Role: FOLLOWER
I20260928 21:53:35.075932 24813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77eb1f982e7c407eba9c019ad9406719" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43303 } }
I20260928 21:53:35.076006 24813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:35.076037 24813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:35.076083 24813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:35.076632 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 24745
I20260928 21:53:35.076732 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:35.076910 3075 external_mini_cluster.cc:1428] Setting key 5be9e353dfbc55ee63eb2134280795de
I20260928 21:53:35.076974 24813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77eb1f982e7c407eba9c019ad9406719" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43303 } }
I20260928 21:53:35.077081 24813 leader_election.cc:302] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77eb1f982e7c407eba9c019ad9406719; no voters:
I20260928 21:53:35.077243 24813 leader_election.cc:288] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:35.077308 24816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:35.077445 24816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [term 1 LEADER]: Becoming Leader. State: Replica: 77eb1f982e7c407eba9c019ad9406719, State: Running, Role: LEADER
I20260928 21:53:35.077478 24813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:35.078189 24816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77eb1f982e7c407eba9c019ad9406719" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43303 } }
I20260928 21:53:35.078799 24819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77eb1f982e7c407eba9c019ad9406719. Latest consensus state: current_term: 1 leader_uuid: "77eb1f982e7c407eba9c019ad9406719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77eb1f982e7c407eba9c019ad9406719" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43303 } } }
I20260928 21:53:35.079077 24819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:35.078800 24817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77eb1f982e7c407eba9c019ad9406719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77eb1f982e7c407eba9c019ad9406719" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 43303 } } }
I20260928 21:53:35.079265 24833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:35.079273 24817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:35.079748 24833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:35.081152 24833 catalog_manager.cc:1347] Generated new cluster ID: 1d0c30f290d04770a414029936b92041
I20260928 21:53:35.081203 24833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:35.082969 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:35.077428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55889 (local address 127.3.0.254:43303)
0928 21:53:35.077716 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:35.077726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:35.077757 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:53:35.077850 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:35.077855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:35.077897 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:35.078020 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:35.078356 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:35.079804 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:35.080907 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:35.081488 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:35.081588 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:35.082318 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:35.082322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:35.082322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:35.082523 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:35.082559 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:35.082573 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:35.082650 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:35.082743 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 21:53:35.082769 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
I20260928 21:53:35.083891 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:35.088972 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:35.085291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34359 (local address 127.3.0.254:43303)
0928 21:53:35.085327 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:35.085329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:35.085430 (+ 101us) server_negotiation.cc:407] Connection header received
0928 21:53:35.085496 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:35.085497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:35.085505 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:35.085559 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:35.085690 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:35.086694 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:35.087684 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:35.087977 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:35.088029 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:35.088740 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:35.088741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:35.088742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:35.088746 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:35.088761 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:35.088765 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:35.088818 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:35.088886 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 21:53:35.088898 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:53:35.090344 24833 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:35.090857 24833 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:35.096321 24833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77eb1f982e7c407eba9c019ad9406719: Generated new TSK 0
I20260928 21:53:35.096529 24833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:35.101634 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 24745
W20260928 21:53:35.105257 24730 connection.cc:535] client connection to 127.3.0.254:43303 recv error: Network error: failed to read from TLS socket (remote: 127.3.0.254:43303): Connection reset by peer (error 104)
2026-09-28T21:53:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:35 dist-test-slave-jrck krb5kdc[24844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:35 dist-test-slave-jrck krb5kdc[24844](info): set up 2 sockets
Sep 28 21:53:35 dist-test-slave-jrck krb5kdc[24844](info): commencing operation
krb5kdc: starting...
W20260928 21:53:36.040304 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:36 dist-test-slave-jrck krb5kdc[24844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:36.153151 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:40761
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:46585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:40761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:36.233707 24857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:36.233945 24857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:36.233989 24857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:36.235221 24857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:36.235270 24857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:36.235289 24857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:36.235306 24857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:36.235321 24857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:36.236893 24857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:46585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:40761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:40761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:36.237156 24857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:36.237376 24857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:36.242036 24863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:36.242072 24865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:36.242096 24862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:36.242213 24857 server_base.cc:1034] running on GCE node
I20260928 21:53:36.242508 24857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:36.242820 24857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:36.243953 24857 hybrid_clock.cc:648] HybridClock initialized: now 1790632416243940 us; error 33 us; skew 500 ppm
Sep 28 21:53:36 dist-test-slave-jrck krb5kdc[24844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632416, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:36.246254 24857 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:36.246683 24857 webserver.cc:460] Webserver started at http://127.3.0.254:36263/ using document root <none> and password file <none>
I20260928 21:53:36.246876 24857 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:36.246923 24857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:36.247059 24857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:36.248076 24857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "0e1aa38e8e9646178acb3f6fb157f1b0"
format_stamp: "Formatted at 2026-09-28 21:53:36 on dist-test-slave-jrck"
server_key: "3992f57e2883f613dfad644694bd9038"
server_key_iv: "acfcda195c22b01398714aa3fe9b4a68"
server_key_version: "encryptionkey@0"
I20260928 21:53:36.248394 24857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "0e1aa38e8e9646178acb3f6fb157f1b0"
format_stamp: "Formatted at 2026-09-28 21:53:36 on dist-test-slave-jrck"
server_key: "3992f57e2883f613dfad644694bd9038"
server_key_iv: "acfcda195c22b01398714aa3fe9b4a68"
server_key_version: "encryptionkey@0"
I20260928 21:53:36.249636 24857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:53:36.250288 24872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:36.250430 24857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 21:53:36.250516 24857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "0e1aa38e8e9646178acb3f6fb157f1b0"
format_stamp: "Formatted at 2026-09-28 21:53:36 on dist-test-slave-jrck"
server_key: "3992f57e2883f613dfad644694bd9038"
server_key_iv: "acfcda195c22b01398714aa3fe9b4a68"
server_key_version: "encryptionkey@0"
I20260928 21:53:36.250572 24857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:36.253775 24857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:36.255466 24857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:36.255589 24857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:36.259208 24857 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:40761
I20260928 21:53:36.259253 24924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:40761 every 8 connection(s)
I20260928 21:53:36.259543 24857 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:36.260507 24925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:36.262718 24925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0: Bootstrap starting.
I20260928 21:53:36.263267 24925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:36.263526 24925 log.cc:824] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:36.264148 24925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0: No bootstrap required, opened a new log
I20260928 21:53:36.265797 24925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1aa38e8e9646178acb3f6fb157f1b0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40761 } }
I20260928 21:53:36.265946 24925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:36.265969 24925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1aa38e8e9646178acb3f6fb157f1b0, State: Initialized, Role: FOLLOWER
I20260928 21:53:36.266100 24925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1aa38e8e9646178acb3f6fb157f1b0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40761 } }
I20260928 21:53:36.266162 24925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:36.266183 24925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:36.266230 24925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:36.267053 24925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1aa38e8e9646178acb3f6fb157f1b0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40761 } }
I20260928 21:53:36.267143 24925 leader_election.cc:302] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e1aa38e8e9646178acb3f6fb157f1b0; no voters:
I20260928 21:53:36.267315 24925 leader_election.cc:288] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:36.267381 24928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:36.267496 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 24857
I20260928 21:53:36.267524 24928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [term 1 LEADER]: Becoming Leader. State: Replica: 0e1aa38e8e9646178acb3f6fb157f1b0, State: Running, Role: LEADER
I20260928 21:53:36.267570 24925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:36.267637 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:36.267606 24928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1aa38e8e9646178acb3f6fb157f1b0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40761 } }
I20260928 21:53:36.267844 3075 external_mini_cluster.cc:1428] Setting key 13b8df5402a9dc39f5874e6cbe97ba12
I20260928 21:53:36.267993 24929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e1aa38e8e9646178acb3f6fb157f1b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1aa38e8e9646178acb3f6fb157f1b0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40761 } } }
I20260928 21:53:36.268127 24929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:36.268378 24937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:36.268551 24928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e1aa38e8e9646178acb3f6fb157f1b0. Latest consensus state: current_term: 1 leader_uuid: "0e1aa38e8e9646178acb3f6fb157f1b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1aa38e8e9646178acb3f6fb157f1b0" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 40761 } } }
I20260928 21:53:36.269002 24928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:36.269825 24937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:36.270390 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:36.268502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45873 (local address 127.3.0.254:40761)
0928 21:53:36.268978 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:36.268988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:36.269023 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:53:36.269140 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:36.269148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:36.269180 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:36.269330 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:36.269946 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:36.269950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:36.269950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:36.269954 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:36.269999 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:36.270015 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:36.270074 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:36.270178 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 21:53:36.270219 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":80,"threads_started":1}
I20260928 21:53:36.271288 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:36.271468 24937 catalog_manager.cc:1347] Generated new cluster ID: a0c7c1b26a65485fb1c37708368c9fa0
I20260928 21:53:36.271514 24937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:36.273834 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:36.272797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36111 (local address 127.3.0.254:40761)
0928 21:53:36.272829 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:36.272831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:36.272900 (+ 69us) server_negotiation.cc:407] Connection header received
0928 21:53:36.272983 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:36.272984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:36.272988 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:36.273051 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:36.273618 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:36.273619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:36.273620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:36.273624 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:36.273641 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:36.273645 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:36.273701 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:36.273747 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 21:53:36.273761 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:53:36.287577 24937 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:36.287777 24937 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:36.295694 24937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e1aa38e8e9646178acb3f6fb157f1b0: Generated new TSK 0
I20260928 21:53:36.295899 24937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:36.304109 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 24857
2026-09-28T21:53:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:36 dist-test-slave-jrck krb5kdc[24956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:36 dist-test-slave-jrck krb5kdc[24956](info): set up 2 sockets
Sep 28 21:53:36 dist-test-slave-jrck krb5kdc[24956](info): commencing operation
krb5kdc: starting...
W20260928 21:53:37.257246 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:37 dist-test-slave-jrck krb5kdc[24956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:37.373445 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:44107
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:37.455077 24969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:37.455313 24969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:37.455348 24969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:37.456600 24969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:37.456645 24969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:37.456661 24969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:37.456677 24969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:37.456692 24969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:37.458499 24969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:44107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:44107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:37.458786 24969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:37.459003 24969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:37.463832 24975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:37.463847 24969 server_base.cc:1034] running on GCE node
W20260928 21:53:37.463914 24974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:37.463838 24977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:37.464277 24969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:37.464618 24969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:37.465771 24969 hybrid_clock.cc:648] HybridClock initialized: now 1790632417465746 us; error 37 us; skew 500 ppm
Sep 28 21:53:37 dist-test-slave-jrck krb5kdc[24956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632417, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:37.471541 24969 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:37.472012 24969 webserver.cc:460] Webserver started at http://127.3.0.254:43627/ using document root <none> and password file <none>
I20260928 21:53:37.472213 24969 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:37.472251 24969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:37.472344 24969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:37.473524 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "5fe0177e03144ad4b80f5914fa73ad05"
format_stamp: "Formatted at 2026-09-28 21:53:37 on dist-test-slave-jrck"
server_key: "d3ad8faea16a7a29ed9aaeae721e22d7"
server_key_iv: "8739e2aa9c914589f500c7731d3e7363"
server_key_version: "encryptionkey@0"
I20260928 21:53:37.473827 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "5fe0177e03144ad4b80f5914fa73ad05"
format_stamp: "Formatted at 2026-09-28 21:53:37 on dist-test-slave-jrck"
server_key: "d3ad8faea16a7a29ed9aaeae721e22d7"
server_key_iv: "8739e2aa9c914589f500c7731d3e7363"
server_key_version: "encryptionkey@0"
I20260928 21:53:37.475076 24969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:53:37.475833 24984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:37.476023 24969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:37.476164 24969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "5fe0177e03144ad4b80f5914fa73ad05"
format_stamp: "Formatted at 2026-09-28 21:53:37 on dist-test-slave-jrck"
server_key: "d3ad8faea16a7a29ed9aaeae721e22d7"
server_key_iv: "8739e2aa9c914589f500c7731d3e7363"
server_key_version: "encryptionkey@0"
I20260928 21:53:37.476240 24969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:37.485917 24969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:37.488274 24969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:37.488390 24969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:37.492102 24969 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:44107
I20260928 21:53:37.492137 25036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:44107 every 8 connection(s)
I20260928 21:53:37.492535 24969 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:37.493417 25037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:37.495711 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05: Bootstrap starting.
I20260928 21:53:37.496273 25037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:37.496575 25037 log.cc:824] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:37.497259 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05: No bootstrap required, opened a new log
I20260928 21:53:37.498540 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 24969
I20260928 21:53:37.498627 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:37.498783 3075 external_mini_cluster.cc:1428] Setting key f987a5848b405003c7b08484583408fd
I20260928 21:53:37.498907 25037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0177e03144ad4b80f5914fa73ad05" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44107 } }
I20260928 21:53:37.499060 25037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:37.499115 25037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe0177e03144ad4b80f5914fa73ad05, State: Initialized, Role: FOLLOWER
I20260928 21:53:37.499228 25037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0177e03144ad4b80f5914fa73ad05" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44107 } }
I20260928 21:53:37.499312 25037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:37.499351 25037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:37.499393 25037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:37.500443 25037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0177e03144ad4b80f5914fa73ad05" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44107 } }
I20260928 21:53:37.500603 25037 leader_election.cc:302] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fe0177e03144ad4b80f5914fa73ad05; no voters:
I20260928 21:53:37.500754 25037 leader_election.cc:288] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:37.500803 25042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:37.500910 25042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [term 1 LEADER]: Becoming Leader. State: Replica: 5fe0177e03144ad4b80f5914fa73ad05, State: Running, Role: LEADER
I20260928 21:53:37.500989 25042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0177e03144ad4b80f5914fa73ad05" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44107 } }
I20260928 21:53:37.501037 25037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:37.501299 25042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fe0177e03144ad4b80f5914fa73ad05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0177e03144ad4b80f5914fa73ad05" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44107 } } }
I20260928 21:53:37.501355 25042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:37.501456 25042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fe0177e03144ad4b80f5914fa73ad05. Latest consensus state: current_term: 1 leader_uuid: "5fe0177e03144ad4b80f5914fa73ad05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fe0177e03144ad4b80f5914fa73ad05" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44107 } } }
I20260928 21:53:37.501500 25042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05 [sys.catalog]: This master's current role is: LEADER
W20260928 21:53:37.502655 25057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:53:37.503021 25057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:53:37.503103 25047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:37.503283 25047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:37.504971 25047 catalog_manager.cc:1347] Generated new cluster ID: 9c6bfb2a87234dea92ff778591e70007
I20260928 21:53:37.505025 25047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 21:53:37 dist-test-slave-jrck krb5kdc[24956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:37.511000 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:37.499264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44651 (local address 127.3.0.254:44107)
0928 21:53:37.499530 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:37.499571 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:37.500527 (+ 956us) server_negotiation.cc:407] Connection header received
0928 21:53:37.500704 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:37.500712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:37.500766 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:37.500940 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:37.501647 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:37.502839 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:37.503651 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:37.504220 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:37.504337 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:37.507128 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:37.507133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:37.507133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:37.507147 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:37.509233 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:37.509633 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:37.509637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:37.509638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:37.509777 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:37.509960 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:37.509961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:37.509962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:37.510313 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:37.510474 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:37.510598 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 21:53:37.510758 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260928 21:53:37.512084 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:37.517678 25047 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:37.518384 25047 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:37.524075 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:37.513654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38041 (local address 127.3.0.254:44107)
0928 21:53:37.513690 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:37.513692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:37.516694 (+ 3002us) server_negotiation.cc:407] Connection header received
0928 21:53:37.516761 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:37.516762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:37.516775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:37.516848 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:37.517590 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:37.518323 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:37.519432 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:37.519763 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:37.519833 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:37.521366 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:37.521368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:37.521368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:37.521376 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:37.522916 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:37.523317 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:37.523319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:37.523319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:37.523374 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:37.523557 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:37.523558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:37.523558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:37.523674 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:37.523760 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:37.523875 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 21:53:37.523972 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:53:37.527521 25047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fe0177e03144ad4b80f5914fa73ad05: Generated new TSK 0
I20260928 21:53:37.527740 25047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:37.530494 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 24969
2026-09-28T21:53:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:37 dist-test-slave-jrck krb5kdc[25068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:37 dist-test-slave-jrck krb5kdc[25068](info): set up 2 sockets
Sep 28 21:53:37 dist-test-slave-jrck krb5kdc[25068](info): commencing operation
krb5kdc: starting...
W20260928 21:53:38.488668 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:38 dist-test-slave-jrck krb5kdc[25068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:38.621248 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:44525
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:32797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:44525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:38.707288 25081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:38.707533 25081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:38.707568 25081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:38.708866 25081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:38.708920 25081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:38.708940 25081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:38.708957 25081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:38.708971 25081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:38.710588 25081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:32797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:44525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:38.711146 25081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:38.711511 25081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:53:38.716753 25081 server_base.cc:1034] running on GCE node
W20260928 21:53:38.716720 25086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:38.716719 25089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:38.716732 25087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:38.717273 25081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:38.717656 25081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:38.718863 25081 hybrid_clock.cc:648] HybridClock initialized: now 1790632418718824 us; error 60 us; skew 500 ppm
Sep 28 21:53:38 dist-test-slave-jrck krb5kdc[25068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632418, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:38.721819 25081 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:38.722352 25081 webserver.cc:460] Webserver started at http://127.3.0.254:45575/ using document root <none> and password file <none>
I20260928 21:53:38.722563 25081 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:38.722606 25081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:38.722707 25081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:38.723800 25081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "10cbc6ee34df4d19b3f489551a35e5d5"
format_stamp: "Formatted at 2026-09-28 21:53:38 on dist-test-slave-jrck"
server_key: "4638df4b8e97cf983bc34957c336f249"
server_key_iv: "cc1fce1f2895aeacae10242e8ce96c01"
server_key_version: "encryptionkey@0"
I20260928 21:53:38.724131 25081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "10cbc6ee34df4d19b3f489551a35e5d5"
format_stamp: "Formatted at 2026-09-28 21:53:38 on dist-test-slave-jrck"
server_key: "4638df4b8e97cf983bc34957c336f249"
server_key_iv: "cc1fce1f2895aeacae10242e8ce96c01"
server_key_version: "encryptionkey@0"
I20260928 21:53:38.725582 25081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:38.726437 25096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:38.726771 25081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:53:38.726862 25081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "10cbc6ee34df4d19b3f489551a35e5d5"
format_stamp: "Formatted at 2026-09-28 21:53:38 on dist-test-slave-jrck"
server_key: "4638df4b8e97cf983bc34957c336f249"
server_key_iv: "cc1fce1f2895aeacae10242e8ce96c01"
server_key_version: "encryptionkey@0"
I20260928 21:53:38.726926 25081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:38.737970 25081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:38.740181 25081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:38.740307 25081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:38.744443 25081 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:44525
I20260928 21:53:38.744592 25148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:44525 every 8 connection(s)
I20260928 21:53:38.744894 25081 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:38.745851 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:38.746707 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 25081
I20260928 21:53:38.746810 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:38.746984 3075 external_mini_cluster.cc:1428] Setting key 6c12f561a4bde5b211e9637de91cd863
I20260928 21:53:38.748584 25149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5: Bootstrap starting.
I20260928 21:53:38.749194 25149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:38.749468 25149 log.cc:824] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:38.750198 25149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5: No bootstrap required, opened a new log
I20260928 21:53:38.752007 25149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cbc6ee34df4d19b3f489551a35e5d5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44525 } }
I20260928 21:53:38.752154 25149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:38.752187 25149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10cbc6ee34df4d19b3f489551a35e5d5, State: Initialized, Role: FOLLOWER
I20260928 21:53:38.752354 25149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cbc6ee34df4d19b3f489551a35e5d5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44525 } }
I20260928 21:53:38.752481 25149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:38.752552 25149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:38.752607 25149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:38.752748 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:38.747497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34339 (local address 127.3.0.254:44525)
0928 21:53:38.747755 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:38.747763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:38.747807 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:53:38.748331 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:38.748339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:38.748406 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:38.748623 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:38.748855 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:38.749961 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:38.750813 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:38.751261 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:38.751361 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:38.752010 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:38.752015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:38.752016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:38.752187 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:38.752226 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:38.752243 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:38.752307 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:38.752407 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 21:53:38.752496 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260928 21:53:38.753350 25149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cbc6ee34df4d19b3f489551a35e5d5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44525 } }
I20260928 21:53:38.753449 25149 leader_election.cc:302] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10cbc6ee34df4d19b3f489551a35e5d5; no voters:
I20260928 21:53:38.753618 25149 leader_election.cc:288] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:38.753690 25154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:38.753731 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:38.753911 25154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [term 1 LEADER]: Becoming Leader. State: Replica: 10cbc6ee34df4d19b3f489551a35e5d5, State: Running, Role: LEADER
I20260928 21:53:38.753934 25149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:38.755502 25154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cbc6ee34df4d19b3f489551a35e5d5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44525 } }
I20260928 21:53:38.756093 25155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10cbc6ee34df4d19b3f489551a35e5d5. Latest consensus state: current_term: 1 leader_uuid: "10cbc6ee34df4d19b3f489551a35e5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cbc6ee34df4d19b3f489551a35e5d5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44525 } } }
I20260928 21:53:38.756148 25156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10cbc6ee34df4d19b3f489551a35e5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cbc6ee34df4d19b3f489551a35e5d5" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44525 } } }
I20260928 21:53:38.756299 25155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:38.756304 25156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5 [sys.catalog]: This master's current role is: LEADER
W20260928 21:53:38.756882 25173 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:53:38.756942 25173 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:53:38.757501 25175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:38.757694 25175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:38.758765 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:38.755434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35545 (local address 127.3.0.254:44525)
0928 21:53:38.755470 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:38.755472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:38.755575 (+ 103us) server_negotiation.cc:407] Connection header received
0928 21:53:38.755618 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:38.755619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:38.755628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:38.755688 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:38.755811 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:38.756547 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:38.757407 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:38.757709 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:38.757783 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:38.758540 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:38.758541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:38.758541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:38.758546 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:38.758563 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:38.758567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:38.758612 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:38.758665 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 21:53:38.758695 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:53:38.759269 25175 catalog_manager.cc:1347] Generated new cluster ID: 0f3dbb2bd1fb4321a3b95a25bd1b5491
I20260928 21:53:38.759330 25175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:38.768929 25175 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:38.769585 25175 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:38.778136 25175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10cbc6ee34df4d19b3f489551a35e5d5: Generated new TSK 0
I20260928 21:53:38.778414 25175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:38.782732 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 25081
2026-09-28T21:53:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:38 dist-test-slave-jrck krb5kdc[25180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:38 dist-test-slave-jrck krb5kdc[25180](info): set up 2 sockets
Sep 28 21:53:38 dist-test-slave-jrck krb5kdc[25180](info): commencing operation
krb5kdc: starting...
W20260928 21:53:39.744551 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:39 dist-test-slave-jrck krb5kdc[25180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:39.859123 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:37173
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:45615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:37173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:39.939671 25193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:39.939909 25193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:39.939941 25193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:39.941196 25193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:39.941254 25193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:39.941273 25193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:39.941290 25193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:39.941305 25193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:39.942852 25193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:45615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:37173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:37173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:39.943107 25193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:39.943325 25193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:39.947973 25199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:39.947976 25198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:39.947973 25201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:39.948338 25193 server_base.cc:1034] running on GCE node
I20260928 21:53:39.948606 25193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:39.948884 25193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:39.950032 25193 hybrid_clock.cc:648] HybridClock initialized: now 1790632419950013 us; error 30 us; skew 500 ppm
Sep 28 21:53:39 dist-test-slave-jrck krb5kdc[25180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632419, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:39.952492 25193 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:39.952932 25193 webserver.cc:460] Webserver started at http://127.3.0.254:45615/ using document root <none> and password file <none>
I20260928 21:53:39.953131 25193 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:39.953174 25193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:39.953295 25193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:39.954444 25193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "2aec25c9139c43b099dfef1327bf7578"
format_stamp: "Formatted at 2026-09-28 21:53:39 on dist-test-slave-jrck"
server_key: "bc106d609712c34e993dca5e794e62e1"
server_key_iv: "695aa9379e68e9f80a45b24aad990370"
server_key_version: "encryptionkey@0"
I20260928 21:53:39.954775 25193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "2aec25c9139c43b099dfef1327bf7578"
format_stamp: "Formatted at 2026-09-28 21:53:39 on dist-test-slave-jrck"
server_key: "bc106d609712c34e993dca5e794e62e1"
server_key_iv: "695aa9379e68e9f80a45b24aad990370"
server_key_version: "encryptionkey@0"
I20260928 21:53:39.956017 25193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:53:39.956734 25208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:39.956883 25193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:39.956970 25193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "2aec25c9139c43b099dfef1327bf7578"
format_stamp: "Formatted at 2026-09-28 21:53:39 on dist-test-slave-jrck"
server_key: "bc106d609712c34e993dca5e794e62e1"
server_key_iv: "695aa9379e68e9f80a45b24aad990370"
server_key_version: "encryptionkey@0"
I20260928 21:53:39.957034 25193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:39.960282 25193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:39.962172 25193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:39.962284 25193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:39.966058 25193 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:37173
I20260928 21:53:39.966109 25260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:37173 every 8 connection(s)
I20260928 21:53:39.966409 25193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:39.967382 25261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:39.969789 25261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578: Bootstrap starting.
I20260928 21:53:39.970353 25261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:39.970611 25261 log.cc:824] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:39.971452 25261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578: No bootstrap required, opened a new log
I20260928 21:53:39.973124 25261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aec25c9139c43b099dfef1327bf7578" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37173 } }
I20260928 21:53:39.973281 25261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:53:39.973313 25261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aec25c9139c43b099dfef1327bf7578, State: Initialized, Role: FOLLOWER
I20260928 21:53:39.973461 25261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aec25c9139c43b099dfef1327bf7578" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37173 } }
I20260928 21:53:39.973539 25261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:53:39.973579 25261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:53:39.973627 25261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:39.973778 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 25193
I20260928 21:53:39.973868 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:39.974030 3075 external_mini_cluster.cc:1428] Setting key 963a474abd38e964b317e074536448cb
I20260928 21:53:39.974707 25261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aec25c9139c43b099dfef1327bf7578" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37173 } }
I20260928 21:53:39.974817 25261 leader_election.cc:302] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aec25c9139c43b099dfef1327bf7578; no voters:
I20260928 21:53:39.974967 25261 leader_election.cc:288] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:53:39.974993 25266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:39.975116 25266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [term 1 LEADER]: Becoming Leader. State: Replica: 2aec25c9139c43b099dfef1327bf7578, State: Running, Role: LEADER
I20260928 21:53:39.975212 25261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:39.975208 25266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aec25c9139c43b099dfef1327bf7578" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37173 } }
I20260928 21:53:39.975584 25268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aec25c9139c43b099dfef1327bf7578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aec25c9139c43b099dfef1327bf7578" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37173 } } }
I20260928 21:53:39.975730 25268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:39.976514 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:39.974554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53127 (local address 127.3.0.254:37173)
0928 21:53:39.974753 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:39.974764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:39.974801 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:53:39.974938 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:39.974946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:39.974980 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:39.975131 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:39.975954 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:39.975959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:39.975959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:39.975964 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:39.976017 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:39.976034 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:39.976145 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:39.976224 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 21:53:39.976274 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260928 21:53:39.977137 25267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aec25c9139c43b099dfef1327bf7578. Latest consensus state: current_term: 1 leader_uuid: "2aec25c9139c43b099dfef1327bf7578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aec25c9139c43b099dfef1327bf7578" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 37173 } } }
I20260928 21:53:39.977219 25267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:39.977596 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 21:53:39.977679 25282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:53:39.977730 25282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:53:39.977779 25272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:39.977931 25272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:39.980127 25272 catalog_manager.cc:1347] Generated new cluster ID: 72a311e128094ef0a0644895184b3838
I20260928 21:53:39.980196 25272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:39.980320 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:39.979156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54989 (local address 127.3.0.254:37173)
0928 21:53:39.979190 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:39.979192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:39.979259 (+ 67us) server_negotiation.cc:407] Connection header received
0928 21:53:39.979337 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:39.979338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:39.979343 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:39.979397 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:39.979987 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:39.979988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:39.979988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:53:39.979992 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:39.980008 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:53:39.980012 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:39.980058 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:39.980124 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 21:53:39.980140 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 21:53:40.061939 25272 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:40.062122 25272 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:40.070724 25272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aec25c9139c43b099dfef1327bf7578: Generated new TSK 0
I20260928 21:53:40.070930 25272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:40.075801 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 25193
2026-09-28T21:53:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1292 ms)
[----------] 18 tests from AuthTokenIssuingTest (22073 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:40 dist-test-slave-jrck krb5kdc[25292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:40 dist-test-slave-jrck krb5kdc[25292](info): set up 2 sockets
Sep 28 21:53:40 dist-test-slave-jrck krb5kdc[25292](info): commencing operation
krb5kdc: starting...
W20260928 21:53:41.025048 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:41.141682 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:42471
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:42471,127.3.0.253:43473,127.3.0.252:42369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:41.223601 25305 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:53:41.223883 25305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:41.223910 25305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:41.223940 25305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:41.225317 25305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:41.225378 25305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:41.225399 25305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:41.225414 25305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:41.225430 25305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:41.227119 25305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:42471,127.3.0.253:43473,127.3.0.252:42369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:42471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:41.227388 25305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:41.227617 25305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:41.232932 25313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:41.232936 25310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:41.232919 25305 server_base.cc:1034] running on GCE node
W20260928 21:53:41.232919 25311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:41.233361 25305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:41.233657 25305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:41.234822 25305 hybrid_clock.cc:648] HybridClock initialized: now 1790632421234805 us; error 33 us; skew 500 ppm
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:41.237510 25305 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:41.238026 25305 webserver.cc:460] Webserver started at http://127.3.0.254:40277/ using document root <none> and password file <none>
I20260928 21:53:41.238253 25305 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:41.238300 25305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:41.238423 25305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:41.239516 25305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "c813d72290d34213a42f9a08eeeb3e39"
format_stamp: "Formatted at 2026-09-28 21:53:41 on dist-test-slave-jrck"
server_key: "90bf8455207444d7a6c9bb57ee9a18ab"
server_key_iv: "f998eed02b66a018833b909430baae41"
server_key_version: "encryptionkey@0"
I20260928 21:53:41.239858 25305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "c813d72290d34213a42f9a08eeeb3e39"
format_stamp: "Formatted at 2026-09-28 21:53:41 on dist-test-slave-jrck"
server_key: "90bf8455207444d7a6c9bb57ee9a18ab"
server_key_iv: "f998eed02b66a018833b909430baae41"
server_key_version: "encryptionkey@0"
I20260928 21:53:41.241230 25305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 21:53:41.242071 25320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:41.242237 25305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:41.242331 25305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "c813d72290d34213a42f9a08eeeb3e39"
format_stamp: "Formatted at 2026-09-28 21:53:41 on dist-test-slave-jrck"
server_key: "90bf8455207444d7a6c9bb57ee9a18ab"
server_key_iv: "f998eed02b66a018833b909430baae41"
server_key_version: "encryptionkey@0"
I20260928 21:53:41.242409 25305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:41.261790 25305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:41.264225 25305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:41.264371 25305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:41.268399 25305 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:42471
I20260928 21:53:41.268445 25372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:42471 every 8 connection(s)
I20260928 21:53:41.268822 25305 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:41.269801 25373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:41.271397 25373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:41.276754 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 25305
I20260928 21:53:41.276865 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:41.277036 3075 external_mini_cluster.cc:1428] Setting key ba95ae7f0a5e6efd8ce3917dc4b03281
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:41.281765 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.272118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:42471 (local address 127.0.0.1:46606)
0928 21:53:41.272303 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.272347 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:41.272895 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:41.273213 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:41.273216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:41.273492 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:41.273688 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.273694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.274789 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.274790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.275515 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.275517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.276040 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.276041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.276095 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.276688 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:41.276696 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:41.278256 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:41.280043 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.280050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.280052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.280265 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.280539 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.280540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.280540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.280639 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.281118 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:41.281124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:41.281199 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 21:53:41.281283 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:53:41.281541 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":89,"thread_start_us":74,"threads_started":1}
I20260928 21:53:41.281771 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.272196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.3.0.254:42471)
0928 21:53:41.272348 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.272358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.272424 (+ 66us) server_negotiation.cc:407] Connection header received
0928 21:53:41.273086 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.273088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.273106 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.273201 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.273797 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.274683 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.275611 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.275954 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.276049 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.278345 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.278352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.278353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.278360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.279952 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.280342 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.280346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.280346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.280479 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.280711 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.280713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.280713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.281020 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.281117 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.281329 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.281473 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260928 21:53:41.282666 25373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:41.283066 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.282918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:43473 (local address 127.0.0.1:53108)
0928 21:53:41.282991 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.283040 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:43473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 21:53:41.283207 25321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.253:43473: connect: Connection refused (error 111)
I20260928 21:53:41.283648 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.283555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:43473 (local address 127.0.0.1:53112)
0928 21:53:41.283600 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.283627 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:43473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 21:53:41.283746 25373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.253:43473: Network error: Client connection negotiation failed: client connection to 127.3.0.253:43473: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.0.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab.
Entry for principal kudu/127.3.0.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab.
I20260928 21:53:41.303061 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.253:43473
--webserver_interface=127.3.0.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab
--principal=kudu/127.3.0.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:42471,127.3.0.253:43473,127.3.0.252:42369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:53:41.333014 25373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } attempt: 1
I20260928 21:53:41.333523 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.333369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:43473 (local address 127.0.0.1:53120)
0928 21:53:41.333448 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.333488 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:43473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:53:41.333849 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.333768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:43473 (local address 127.0.0.1:53124)
0928 21:53:41.333802 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.333828 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:43473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 21:53:41.333966 25373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.253:43473: Network error: Client connection negotiation failed: client connection to 127.3.0.253:43473: connect: Connection refused (error 111)
W20260928 21:53:41.387372 25380 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:53:41.387612 25380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:41.387629 25380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:41.387650 25380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:41.388924 25380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:41.389045 25380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:41.389088 25380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:41.389110 25380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:41.389127 25380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:41.390924 25380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:42471,127.3.0.253:43473,127.3.0.252:42369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab
--principal=kudu/127.3.0.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.253:43473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.0.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:41.391172 25380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:41.391377 25380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:41.396476 25386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:41.396471 25385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:41.396562 25380 server_base.cc:1034] running on GCE node
W20260928 21:53:41.396447 25388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:41.397042 25380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:41.397384 25380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:41.398563 25380 hybrid_clock.cc:648] HybridClock initialized: now 1790632421398548 us; error 31 us; skew 500 ppm
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:41.401423 25380 init.cc:377] Logged in from keytab as kudu/127.3.0.253@KRBTEST.COM (short username kudu)
I20260928 21:53:41.401963 25380 webserver.cc:460] Webserver started at http://127.3.0.253:44047/ using document root <none> and password file <none>
I20260928 21:53:41.402168 25373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } attempt: 2
I20260928 21:53:41.402321 25380 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:41.402395 25380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:41.402535 25380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:41.402523 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.402421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:43473 (local address 127.0.0.1:53136)
0928 21:53:41.402460 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.402493 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:43473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 21:53:41.402894 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.402814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:43473 (local address 127.0.0.1:53152)
0928 21:53:41.402848 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.402873 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:43473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 21:53:41.402968 25373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.253:43473: Network error: Client connection negotiation failed: client connection to 127.3.0.253:43473: connect: Connection refused (error 111)
I20260928 21:53:41.403682 25380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/data/instance:
uuid: "184a7d4acd76439bad9080798555bf3b"
format_stamp: "Formatted at 2026-09-28 21:53:41 on dist-test-slave-jrck"
server_key: "68454a74883a602ef37b62c4f55a2652"
server_key_iv: "7fef94e648595aa9958e6e9922a6a865"
server_key_version: "encryptionkey@0"
I20260928 21:53:41.404039 25380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/wal/instance:
uuid: "184a7d4acd76439bad9080798555bf3b"
format_stamp: "Formatted at 2026-09-28 21:53:41 on dist-test-slave-jrck"
server_key: "68454a74883a602ef37b62c4f55a2652"
server_key_iv: "7fef94e648595aa9958e6e9922a6a865"
server_key_version: "encryptionkey@0"
I20260928 21:53:41.405542 25380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 21:53:41.406350 25395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:41.406529 25380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:53:41.406625 25380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/wal
uuid: "184a7d4acd76439bad9080798555bf3b"
format_stamp: "Formatted at 2026-09-28 21:53:41 on dist-test-slave-jrck"
server_key: "68454a74883a602ef37b62c4f55a2652"
server_key_iv: "7fef94e648595aa9958e6e9922a6a865"
server_key_version: "encryptionkey@0"
I20260928 21:53:41.406695 25380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:41.425354 25380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:41.427601 25380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:41.427744 25380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:41.431769 25380 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.253:43473
I20260928 21:53:41.431818 25447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.253:43473 every 8 connection(s)
I20260928 21:53:41.432168 25380 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
I20260928 21:53:41.433221 25448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:41.434901 25448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:41.438144 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 25380
I20260928 21:53:41.438232 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-1/wal/instance
I20260928 21:53:41.438397 3075 external_mini_cluster.cc:1428] Setting key 426f605ea2104a04d95148eedf700c78
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:41.444679 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.435633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.3.0.254:42471)
0928 21:53:41.435697 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.435699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.435925 (+ 226us) server_negotiation.cc:407] Connection header received
0928 21:53:41.436425 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.436426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.436439 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.436537 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.437201 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.437742 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.438972 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.439280 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.439358 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.441669 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.441671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.441671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.441680 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.443180 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.443598 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.443599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.443600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.443664 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.443902 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.443903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.443903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.444064 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.444134 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.444482 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.444580 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:53:41.444846 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.435611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:42471 (local address 127.0.0.1:46618)
0928 21:53:41.435776 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.435819 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:41.436293 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:41.436605 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:41.436609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:41.436879 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:41.437103 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.437108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.437881 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.437882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.438865 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.438868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.439387 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.439388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.439523 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.440041 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:41.440050 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:41.441565 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:41.443279 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.443286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.443288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.443526 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.443725 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.443726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.443726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.443821 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.444157 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:41.444165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:41.444309 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 21:53:41.444420 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:53:41.444603 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260928 21:53:41.445448 25448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.0.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.252@KRBTEST.COM" created.
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
I20260928 21:53:41.452831 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.445722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:43473 (local address 127.0.0.1:53162)
0928 21:53:41.445762 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.445773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:41.445813 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:41.446150 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:41.446151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:41.446324 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:41.446396 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.446399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.447280 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.447281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.447872 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.447875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.448352 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.448353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.448436 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.448872 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:41.448877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:41.450029 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:41.451838 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.451839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.451839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.451995 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.452178 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.452179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.452179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.452226 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.452496 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:41.452497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:41.452537 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:53:41.452595 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:53:41.452699 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 21:53:41.452915 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.445795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.3.0.253:43473)
0928 21:53:41.445960 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.445965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.445987 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:53:41.446009 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.446012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.446035 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.446114 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.446470 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.447199 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.447942 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.448270 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.448320 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.450105 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.450116 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.450116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.450124 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.451753 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.452073 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.452076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.452076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.452125 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.452275 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.452275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.452276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.452415 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.452512 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.452630 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.452773 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260928 21:53:41.453436 25448 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:41.453781 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.453677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32802)
0928 21:53:41.453714 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.453758 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 21:53:41.453886 25397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
I20260928 21:53:41.454295 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.454209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32816)
0928 21:53:41.454246 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.454271 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 21:53:41.454501 25448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:42369: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab.
Entry for principal kudu/127.3.0.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab.
I20260928 21:53:41.463634 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.252:42369
--webserver_interface=127.3.0.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab
--principal=kudu/127.3.0.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:42471,127.3.0.253:43473,127.3.0.252:42369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:53:41.494098 25373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } attempt: 3
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
I20260928 21:53:41.501451 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.494559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.3.0.253:43473)
0928 21:53:41.494631 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.494634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.494647 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:53:41.494703 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.494704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.494717 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.494777 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.495065 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.495870 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.496541 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.496880 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.496950 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.498845 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.498847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.498847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.498855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.500344 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.500773 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.500774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.500774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.500822 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.501003 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.501004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.501004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.501117 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.501161 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.501320 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.501384 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:53:41.501483 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.494505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:43473 (local address 127.0.0.1:53176)
0928 21:53:41.494580 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.494593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:41.494653 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:41.494785 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:41.494786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:41.494935 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:41.494999 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.495001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.495946 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.495947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.496475 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.496476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.496945 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.496945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.497002 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.497461 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:41.497466 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:41.498744 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:41.500432 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.500434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.500435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.500702 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.500878 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.500879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.500879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.500951 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.501167 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:41.501168 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:41.501222 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 21:53:41.501286 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:53:41.501384 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 21:53:41.501844 25373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:41.502200 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.502099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32820)
0928 21:53:41.502147 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.502179 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 21:53:41.502456 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.502388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32826)
0928 21:53:41.502414 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.502438 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 21:53:41.502540 25373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:42369: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
I20260928 21:53:41.503736 25448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } attempt: 1
I20260928 21:53:41.504078 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.503975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32838)
0928 21:53:41.504018 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.504054 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 21:53:41.504379 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.504284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32844)
0928 21:53:41.504325 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.504353 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:53:41.504483 25448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:42369: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
I20260928 21:53:41.533704 25373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } attempt: 1
I20260928 21:53:41.534226 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.534072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32858)
0928 21:53:41.534144 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.534189 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 21:53:41.534626 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.534525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32872)
0928 21:53:41.534569 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.534600 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 21:53:41.534716 25373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:42369: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
W20260928 21:53:41.546968 25455 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:53:41.547195 25455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:41.547212 25455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:41.547235 25455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:41.548434 25455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:41.548511 25455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:41.548527 25455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:41.548539 25455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:41.548549 25455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:41.550117 25455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:42471,127.3.0.253:43473,127.3.0.252:42369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab
--principal=kudu/127.3.0.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.252:42369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.0.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:41.550365 25455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:41.550589 25455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:41.555325 25460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:41.555298 25463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:41.555298 25461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:41.555481 25455 server_base.cc:1034] running on GCE node
I20260928 21:53:41.555853 25455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:41.556173 25455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:41.557336 25455 hybrid_clock.cc:648] HybridClock initialized: now 1790632421557303 us; error 30 us; skew 500 ppm
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:41.560598 25455 init.cc:377] Logged in from keytab as kudu/127.3.0.252@KRBTEST.COM (short username kudu)
I20260928 21:53:41.561059 25455 webserver.cc:460] Webserver started at http://127.3.0.252:34665/ using document root <none> and password file <none>
I20260928 21:53:41.561264 25455 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:41.561304 25455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:41.561409 25455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:41.562453 25455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/data/instance:
uuid: "59fc65c05c1d49f2a24ee9debed1eadb"
format_stamp: "Formatted at 2026-09-28 21:53:41 on dist-test-slave-jrck"
server_key: "6ccaf25a681f3c2db4c2c67974a49314"
server_key_iv: "1c92a8433860aec84ee194d9b0541b8b"
server_key_version: "encryptionkey@0"
I20260928 21:53:41.562765 25455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/wal/instance:
uuid: "59fc65c05c1d49f2a24ee9debed1eadb"
format_stamp: "Formatted at 2026-09-28 21:53:41 on dist-test-slave-jrck"
server_key: "6ccaf25a681f3c2db4c2c67974a49314"
server_key_iv: "1c92a8433860aec84ee194d9b0541b8b"
server_key_version: "encryptionkey@0"
I20260928 21:53:41.563942 25455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:53:41.564597 25470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:41.564728 25455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:41.564810 25455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/wal
uuid: "59fc65c05c1d49f2a24ee9debed1eadb"
format_stamp: "Formatted at 2026-09-28 21:53:41 on dist-test-slave-jrck"
server_key: "6ccaf25a681f3c2db4c2c67974a49314"
server_key_iv: "1c92a8433860aec84ee194d9b0541b8b"
server_key_version: "encryptionkey@0"
I20260928 21:53:41.564885 25455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:41.572620 25448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } attempt: 2
I20260928 21:53:41.573128 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.572983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32888)
0928 21:53:41.573053 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.573097 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 21:53:41.573521 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.573436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32898)
0928 21:53:41.573477 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.573502 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 21:53:41.573614 25448 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:42369: Network error: Client connection negotiation failed: client connection to 127.3.0.252:42369: connect: Connection refused (error 111)
I20260928 21:53:41.582784 25455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:41.585048 25455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:41.585171 25455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:41.589169 25455 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.252:42369
I20260928 21:53:41.589224 25522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.252:42369 every 8 connection(s)
I20260928 21:53:41.589561 25455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
I20260928 21:53:41.590502 25523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:41.592165 25523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:41.598531 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 25455
I20260928 21:53:41.598625 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790632139349757-3075-0/minicluster-data/master-2/wal/instance
I20260928 21:53:41.598816 3075 external_mini_cluster.cc:1428] Setting key 46e0d870423516079ee8ec535e8eb93e
I20260928 21:53:41.601886 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.592911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.3.0.254:42471)
0928 21:53:41.592974 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.592976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.593199 (+ 223us) server_negotiation.cc:407] Connection header received
0928 21:53:41.593757 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.593758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.593768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.593809 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.594440 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.594918 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.595948 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.596123 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.596170 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.598346 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.598348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.598348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.598355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.600579 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.600960 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.600962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.600962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.601028 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.601248 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.601249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.601249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.601373 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.601431 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.601708 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.601792 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:53:41.602080 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.592868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:42471 (local address 127.0.0.1:46628)
0928 21:53:41.593073 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.593123 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:41.593647 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:41.593878 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:41.593881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:41.594153 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:41.594370 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.594376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.595029 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.595030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.595875 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.595878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.596228 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.596229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.596350 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.596867 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:41.596876 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:41.598250 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:41.600646 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.600653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.600656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.600886 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.601082 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.601083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.601083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.601190 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.601431 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:41.601438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:41.601584 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 21:53:41.601678 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 21:53:41.601843 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260928 21:53:41.602679 25523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
I20260928 21:53:41.607372 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.599233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32900 (local address 127.3.0.252:42369)
0928 21:53:41.599380 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.599385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.599574 (+ 189us) server_negotiation.cc:407] Connection header received
0928 21:53:41.599599 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.599601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.599616 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.599680 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.600412 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.601145 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.601873 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.602343 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.602412 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.604747 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.604755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.604756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.604762 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.606294 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.606607 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.606610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.606611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.606668 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.606827 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.606828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.606829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.606975 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.607112 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.607195 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.607268 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260928 21:53:41.609313 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.602990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:43473 (local address 127.0.0.1:53178)
0928 21:53:41.603023 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.603033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:41.603069 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:41.603221 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:41.603222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:41.603353 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:41.603402 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.603403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.604157 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.604158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.604747 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.604749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.605173 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.605174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.605252 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.605577 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:41.605582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:41.606774 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:41.608359 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.608360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.608361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.608571 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.608747 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.608748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.608748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.608803 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.609018 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:41.609019 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:41.609061 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 21:53:41.609121 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:53:41.609214 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 21:53:41.609314 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.603060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.3.0.253:43473)
0928 21:53:41.603127 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.603129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.603137 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:53:41.603159 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.603160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.603170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.603213 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.603470 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.604077 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.604811 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.605105 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.605170 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.606850 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.606851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.606851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.606859 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.608285 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.608634 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.608635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.608636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.608693 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.608852 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.608853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.608853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.608966 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.609021 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.609166 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.609231 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:53:41.609822 25523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:41.609830 25373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } attempt: 2
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
I20260928 21:53:41.615365 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.608141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.3.0.254:42471)
0928 21:53:41.608191 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.608193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.608202 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:53:41.608229 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.608230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.608240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.608301 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.608753 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.609214 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.610084 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.610360 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.610420 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.612820 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.612821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.612822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.612829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.614331 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.614623 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.614624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.614624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.614679 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.614836 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.614837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.614837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.614971 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.615150 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.615192 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.615264 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:53:41.617079 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.610061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32910)
0928 21:53:41.610221 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.610231 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:41.610269 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:41.610399 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:41.610401 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:41.610565 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:41.610605 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.610607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.611253 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.611254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.611837 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.611840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.612299 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.612300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.612352 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.612747 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:41.612752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:41.614125 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:41.615989 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.615991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.615991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.616192 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.616391 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.616392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.616393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.616487 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.616776 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:41.616777 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:41.616824 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:53:41.616885 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:53:41.616985 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260928 21:53:41.617125 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.610104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.3.0.252:42369)
0928 21:53:41.610139 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.610140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.610314 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:53:41.610331 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.610331 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.610342 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.610386 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.610666 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.611186 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.611918 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.612227 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.612532 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.614211 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.614213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.614213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.614220 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.615900 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.616274 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.616275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.616275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.616326 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.616566 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.616567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.616567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.616695 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.616779 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.616917 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.617047 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":15}
I20260928 21:53:41.618192 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.610180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32922)
0928 21:53:41.610419 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.610426 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:41.610454 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:41.610582 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:41.610583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:41.610722 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:41.610758 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.610760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.611664 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.611664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.612354 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.612356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.612788 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.612789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.612845 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.613806 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:41.613811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:41.614997 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:41.617318 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.617319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.617320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.617507 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.617668 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.617669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.617669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.617710 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.617938 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:41.617938 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:41.617975 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:53:41.618030 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 21:53:41.618112 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260928 21:53:41.618636 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.610192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.3.0.252:42369)
0928 21:53:41.610327 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.610328 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.610494 (+ 166us) server_negotiation.cc:407] Connection header received
0928 21:53:41.610508 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.610508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.610515 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.610567 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.610829 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.611575 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.612423 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.612725 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.612783 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.615099 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.615100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.615101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.615106 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.617246 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.617576 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.617577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.617578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.617621 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.617758 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.617759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.617759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.617890 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.617945 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.618455 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.618533 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260928 21:53:41.619570 25373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39: Bootstrap starting.
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
I20260928 21:53:41.620198 25373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:41.620517 25373 log.cc:824] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:41.621495 25373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39: No bootstrap required, opened a new log
I20260928 21:53:41.621610 25523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb: Bootstrap starting.
I20260928 21:53:41.622215 25523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:41.622498 25523 log.cc:824] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:41.623215 25523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb: No bootstrap required, opened a new log
I20260928 21:53:41.623232 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.616048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.3.0.253:43473)
0928 21:53:41.616106 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.616108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.616114 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:53:41.616132 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.616133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.616140 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.616183 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.616768 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.617462 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.618183 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.618387 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.618441 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.620509 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.620510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.620511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.620519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.622084 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.622470 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.622471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.622472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.622533 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.622736 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.622737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.622737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.622864 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.623028 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.623068 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.623146 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:53:41.623477 25373 raft_consensus.cc:348] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } }
I20260928 21:53:41.623591 25373 raft_consensus.cc:729] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c813d72290d34213a42f9a08eeeb3e39, State: Initialized, Role: FOLLOWER
I20260928 21:53:41.623646 25373 consensus_queue.cc:260] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } }
I20260928 21:53:41.623872 25531 sys_catalog.cc:455] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } } }
I20260928 21:53:41.623890 25373 sys_catalog.cc:564] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:41.623996 25531 sys_catalog.cc:458] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:41.625301 25523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } }
I20260928 21:53:41.625453 25523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fc65c05c1d49f2a24ee9debed1eadb, State: Initialized, Role: FOLLOWER
I20260928 21:53:41.625531 25523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } }
I20260928 21:53:41.625762 25543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } } }
I20260928 21:53:41.625824 25523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:41.625897 25543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:41.625974 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:41.626919 25347 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c813d72290d34213a42f9a08eeeb3e39"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46636
I20260928 21:53:41.627039 25347 raft_consensus.cc:482] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 21:53:41.627084 25347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [term 0 FOLLOWER]: Advancing to term 1
W20260928 21:53:41.627658 25542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:53:41.627702 25542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:53:41.628058 25347 raft_consensus.cc:504] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } }
I20260928 21:53:41.628273 25347 leader_election.cc:288] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [CANDIDATE]: Term 1 election: Requested vote from peers 184a7d4acd76439bad9080798555bf3b (127.3.0.253:43473), 59fc65c05c1d49f2a24ee9debed1eadb (127.3.0.252:42369)
I20260928 21:53:41.626505 25305 master_runner.cc:186] Error getting master registration for 127.3.0.253:43473: OK, instance_id { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" instance_seqno: 1790632421430614 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:53:41.628360 25305 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:53:41.628860 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c813d72290d34213a42f9a08eeeb3e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "59fc65c05c1d49f2a24ee9debed1eadb"
I20260928 21:53:41.628962 25497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:41.630205 25497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c813d72290d34213a42f9a08eeeb3e39 in term 1.
I20260928 21:53:41.630570 25322 leader_election.cc:302] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59fc65c05c1d49f2a24ee9debed1eadb, c813d72290d34213a42f9a08eeeb3e39; no voters:
I20260928 21:53:41.630693 25531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:41.630841 25531 raft_consensus.cc:686] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [term 1 LEADER]: Becoming Leader. State: Replica: c813d72290d34213a42f9a08eeeb3e39, State: Running, Role: LEADER
I20260928 21:53:41.630931 25531 consensus_queue.cc:237] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } }
I20260928 21:53:41.632715 25562 sys_catalog.cc:455] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c813d72290d34213a42f9a08eeeb3e39. Latest consensus state: current_term: 1 leader_uuid: "c813d72290d34213a42f9a08eeeb3e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } } }
I20260928 21:53:41.632874 25562 sys_catalog.cc:458] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:41.631074 25455 master_runner.cc:186] Error getting master registration for 127.3.0.253:43473: OK, instance_id { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" instance_seqno: 1790632421430614 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:53:41.633157 25455 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 21:53:41.633172 25555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:53:41.636605 25555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:53:41.640396 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.630012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.3.0.253:43473)
0928 21:53:41.630064 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.630066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.631323 (+ 1257us) server_negotiation.cc:407] Connection header received
0928 21:53:41.631457 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.631458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.631470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.631500 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.633471 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.633941 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.635537 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.635864 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.635926 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.637566 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.637567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.637568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.637576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.639145 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.639507 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.639508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.639508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.639579 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.639759 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.639760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.639775 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.639907 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.640004 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.640212 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.640299 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:53:41.641292 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.629981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.3.0.254:42471)
0928 21:53:41.630020 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.630021 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.631857 (+ 1836us) server_negotiation.cc:407] Connection header received
0928 21:53:41.632073 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.632075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.632097 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.632126 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.633770 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.634504 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.636502 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.636765 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.636804 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.638190 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.638192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.638192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.638200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.639862 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.640346 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.640347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.640347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.640419 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.640742 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.640743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.640744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.640883 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.640988 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.641123 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.641201 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 21:53:41.646859 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.630763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.3.0.252:42369)
0928 21:53:41.630805 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.630807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.636790 (+ 5983us) server_negotiation.cc:407] Connection header received
0928 21:53:41.636884 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.636886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.636898 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.636955 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.641062 (+ 4107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.641742 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.642406 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.642743 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.642811 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.644258 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.644259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.644260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.644267 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.645846 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.646162 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.646164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.646164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.646221 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.646380 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.646381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.646381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.646496 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.646601 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.646691 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.646771 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 21:53:41.656658 25566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:53:41.656924 25566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:41.659200 25497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [term 1 FOLLOWER]: Refusing update from remote peer c813d72290d34213a42f9a08eeeb3e39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:41.659540 25562 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:41.660974 25543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c813d72290d34213a42f9a08eeeb3e39. Latest consensus state: current_term: 1 leader_uuid: "c813d72290d34213a42f9a08eeeb3e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } } }
I20260928 21:53:41.661082 25543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:41.662135 25562 sys_catalog.cc:455] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c813d72290d34213a42f9a08eeeb3e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } } }
I20260928 21:53:41.662230 25562 sys_catalog.cc:458] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:41.662601 25543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c813d72290d34213a42f9a08eeeb3e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } } }
I20260928 21:53:41.662679 25543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:41.662995 25568 mvcc.cc:205] Tried to move back new op lower bound from 7334430399112470528 to 7334430399001161728. Current Snapshot: MvccSnapshot[applied={T|T < 7334430399112470528}]
I20260928 21:53:41.663102 25567 mvcc.cc:205] Tried to move back new op lower bound from 7334430399112470528 to 7334430399001161728. Current Snapshot: MvccSnapshot[applied={T|T < 7334430399112470528}]
I20260928 21:53:41.663693 25566 catalog_manager.cc:1347] Generated new cluster ID: f1ffd67960cb42c3a0a1e015768ec44d
I20260928 21:53:41.663743 25566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:41.664748 25448 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } attempt: 3
Sep 28 21:53:41 dist-test-slave-jrck krb5kdc[25292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
I20260928 21:53:41.674741 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.665010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:42369 (local address 127.0.0.1:32938)
0928 21:53:41.665099 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:41.665110 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:41.665147 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:41.665458 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:41.665459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:41.665648 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:41.665699 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.665701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.666694 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.666696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.667556 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:41.667559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.669137 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.669138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:41.669232 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.669741 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:41.669745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:41.671454 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:41.673417 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.673418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.673419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.673644 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.673918 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:41.673919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:41.673919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:41.673993 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:41.674397 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:41.674398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:41.674465 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 21:53:41.674524 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:53:41.674628 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 21:53:41.675391 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:41.665231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.3.0.252:42369)
0928 21:53:41.665267 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.665269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.665276 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:53:41.665296 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.665297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.665306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.665347 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.665783 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.666561 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.667707 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:41.668014 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:41.668074 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:41.671570 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:41.671572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:41.671572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:41.671596 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:41.673310 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.673745 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.673747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.673747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.673826 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:41.674065 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:41.674065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:41.674066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:41.674262 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:41.674341 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:41.675195 (+ 854us) server_negotiation.cc:299] Negotiation successful
0928 21:53:41.675298 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 21:53:41.677016 25566 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:41.677413 25448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b: Bootstrap starting.
I20260928 21:53:41.677773 25566 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:41.678171 25448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:41.678471 25448 log.cc:824] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:41.680974 25448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b: No bootstrap required, opened a new log
I20260928 21:53:41.683086 25448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } }
I20260928 21:53:41.683238 25448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 184a7d4acd76439bad9080798555bf3b, State: Initialized, Role: FOLLOWER
I20260928 21:53:41.683317 25448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } }
I20260928 21:53:41.683662 25448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:41.684566 25421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:41.685491 25421 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [term 1 FOLLOWER]: Refusing update from remote peer c813d72290d34213a42f9a08eeeb3e39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:41.685678 25422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c813d72290d34213a42f9a08eeeb3e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "184a7d4acd76439bad9080798555bf3b"
I20260928 21:53:41.686026 25566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39: Generated new TSK 0
I20260928 21:53:41.686226 25566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:41.686416 25422 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c813d72290d34213a42f9a08eeeb3e39 in term 1.
I20260928 21:53:41.686793 25531 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:53:41.690444 25589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:53:41.690583 25589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:53:41.694865 25574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c813d72290d34213a42f9a08eeeb3e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } } }
I20260928 21:53:41.694986 25574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:41.695041 25574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c813d72290d34213a42f9a08eeeb3e39. Latest consensus state: current_term: 1 leader_uuid: "c813d72290d34213a42f9a08eeeb3e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } } }
I20260928 21:53:41.695086 25574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:41.695111 25574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c813d72290d34213a42f9a08eeeb3e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } } }
I20260928 21:53:41.695152 25574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:41.697124 25586 mvcc.cc:205] Tried to move back new op lower bound from 7334430399219712000 to 7334430399001161728. Current Snapshot: MvccSnapshot[applied={T|T < 7334430399112470528}]
W20260928 21:53:41.709251 25376 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.707505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.3.0.254:42471)
0928 21:53:41.707551 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.707553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.708537 (+ 984us) server_negotiation.cc:407] Connection header received
0928 21:53:41.708645 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.708646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.708658 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.708688 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.709172 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46656: BlockingRecv error: recv got EOF from 127.0.0.1:46656 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 21:53:41.710125 25528 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.707892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.3.0.252:42369)
0928 21:53:41.707935 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.707937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.709563 (+ 1626us) server_negotiation.cc:407] Connection header received
0928 21:53:41.709668 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.709669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.709680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.709709 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.710054 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32954: BlockingRecv error: recv got EOF from 127.0.0.1:32954 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 21:53:41.710903 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.708226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.3.0.253:43473)
0928 21:53:41.708264 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.708266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.710349 (+ 2083us) server_negotiation.cc:407] Connection header received
0928 21:53:41.710477 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.710478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.710489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.710518 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.710825 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53216: BlockingRecv error: recv got EOF from 127.0.0.1:53216 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 21:53:41.713480 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.712672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53220 (local address 127.3.0.253:43473)
0928 21:53:41.712710 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.712711 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.712926 (+ 215us) server_negotiation.cc:407] Connection header received
0928 21:53:41.713035 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.713037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.713059 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.713087 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.713426 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53220: BlockingRecv error: recv got EOF from 127.0.0.1:53220 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 21:53:41.716612 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.715803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.3.0.253:43473)
0928 21:53:41.715848 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.715850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.716069 (+ 219us) server_negotiation.cc:407] Connection header received
0928 21:53:41.716160 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.716161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.716171 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.716197 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.716553 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53230: BlockingRecv error: recv got EOF from 127.0.0.1:53230 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 21:53:41.720984 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.720003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.3.0.253:43473)
0928 21:53:41.720047 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.720048 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.720307 (+ 259us) server_negotiation.cc:407] Connection header received
0928 21:53:41.720411 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.720412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.720424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.720523 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.720922 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53242: BlockingRecv error: recv got EOF from 127.0.0.1:53242 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 21:53:41.727402 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.726417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.3.0.253:43473)
0928 21:53:41.726461 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.726464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.726739 (+ 275us) server_negotiation.cc:407] Connection header received
0928 21:53:41.726864 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.726865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.726878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.726911 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.727318 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53244: BlockingRecv error: recv got EOF from 127.0.0.1:53244 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 21:53:41.737973 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.736942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.3.0.253:43473)
0928 21:53:41.736990 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.736993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.737285 (+ 292us) server_negotiation.cc:407] Connection header received
0928 21:53:41.737400 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.737401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.737414 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.737445 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.737907 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53256: BlockingRecv error: recv got EOF from 127.0.0.1:53256 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 21:53:41.756494 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.755454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.3.0.253:43473)
0928 21:53:41.755503 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.755505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.755794 (+ 289us) server_negotiation.cc:407] Connection header received
0928 21:53:41.755921 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.755923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.755936 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.755966 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.756392 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53270: BlockingRecv error: recv got EOF from 127.0.0.1:53270 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 21:53:41.790931 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.789998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.3.0.253:43473)
0928 21:53:41.790044 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.790046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.790317 (+ 271us) server_negotiation.cc:407] Connection header received
0928 21:53:41.790423 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.790424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.790437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.790464 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.790870 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53278: BlockingRecv error: recv got EOF from 127.0.0.1:53278 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 21:53:41.857033 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.856316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.3.0.253:43473)
0928 21:53:41.856370 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.856373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.856428 (+ 55us) server_negotiation.cc:407] Connection header received
0928 21:53:41.856536 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.856537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.856549 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.856637 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.856965 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53292: BlockingRecv error: recv got EOF from 127.0.0.1:53292 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 21:53:41.986838 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:41.986215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.3.0.253:43473)
0928 21:53:41.986284 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:41.986286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:41.986304 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:53:41.986376 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:41.986377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:41.986388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:41.986433 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:41.986772 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53306: BlockingRecv error: recv got EOF from 127.0.0.1:53306 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 21:53:42.244761 25452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:42.244156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.3.0.253:43473)
0928 21:53:42.244208 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:42.244210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:42.244219 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:53:42.244244 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:42.244245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:42.244254 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:42.244305 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:42.244697 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53318: BlockingRecv error: recv got EOF from 127.0.0.1:53318 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:53:42.280568 25648 sys_catalog.cc:455] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c813d72290d34213a42f9a08eeeb3e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c813d72290d34213a42f9a08eeeb3e39" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 42471 } } peers { permanent_uuid: "184a7d4acd76439bad9080798555bf3b" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 43473 } } peers { permanent_uuid: "59fc65c05c1d49f2a24ee9debed1eadb" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 42369 } } }
I20260928 21:53:42.280690 25648 sys_catalog.cc:458] T 00000000000000000000000000000000 P c813d72290d34213a42f9a08eeeb3e39 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:42.637089 25555 catalog_manager.cc:1259] Loaded cluster ID: f1ffd67960cb42c3a0a1e015768ec44d
I20260928 21:53:42.637187 25555 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb: loading cluster ID for follower catalog manager: success
I20260928 21:53:42.637997 25555 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb: acquiring CA information for follower catalog manager: success
I20260928 21:53:42.638291 25555 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59fc65c05c1d49f2a24ee9debed1eadb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:53:42.691087 25589 catalog_manager.cc:1259] Loaded cluster ID: f1ffd67960cb42c3a0a1e015768ec44d
I20260928 21:53:42.691165 25589 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b: loading cluster ID for follower catalog manager: success
I20260928 21:53:42.692013 25589 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b: acquiring CA information for follower catalog manager: success
I20260928 21:53:42.692323 25589 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 184a7d4acd76439bad9080798555bf3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:53:42.760651 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:42.758114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.3.0.253:43473)
0928 21:53:42.758244 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:42.758246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:42.758304 (+ 58us) server_negotiation.cc:407] Connection header received
0928 21:53:42.758387 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:42.758388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:42.758440 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:42.758513 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:42.758622 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:42.759120 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:42.759918 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:42.760125 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:42.760183 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:42.760311 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:42.760424 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:42.760530 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:42.760545 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 21:53:42.760566 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260928 21:53:42.764868 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:42.762502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.3.0.252:42369)
0928 21:53:42.762608 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:42.762610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:42.762619 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:53:42.762661 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:42.762662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:42.762702 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:42.762773 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:42.762884 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:42.763384 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:42.764099 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:42.764303 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:42.764358 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:42.764536 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:42.764645 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:42.764711 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:42.764756 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 21:53:42.764778 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
I20260928 21:53:42.769078 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:42.766416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.3.0.253:43473)
0928 21:53:42.766494 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:42.766496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:42.766661 (+ 165us) server_negotiation.cc:407] Connection header received
0928 21:53:42.766745 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:42.766747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:42.766756 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:42.766802 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:42.766981 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:42.767535 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:42.768264 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:42.768539 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:42.768598 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:42.768783 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:42.768829 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:42.768879 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:42.768970 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:53:42.768998 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:53:42.769537 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:42.766406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.3.0.254:42471)
0928 21:53:42.766568 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:42.766570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:42.766578 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:53:42.766622 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:42.766622 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:42.766684 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:42.766767 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:42.766930 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:42.767477 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:42.768612 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:42.768879 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:42.768930 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:42.769104 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:42.769215 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:42.769270 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:42.769420 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:53:42.769448 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":44,"threads_started":1}
I20260928 21:53:42.770010 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:42.766619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.3.0.252:42369)
0928 21:53:42.766674 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:42.766676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:42.766733 (+ 57us) server_negotiation.cc:407] Connection header received
0928 21:53:42.766803 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:42.766804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:42.766813 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:42.766855 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:42.767629 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:42.768334 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:42.769308 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:42.769562 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:42.769600 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:42.769806 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:42.769838 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:42.769885 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:42.769921 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 21:53:42.769942 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:53:42.775111 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 25305
I20260928 21:53:42.779729 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 25380
I20260928 21:53:42.783757 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 25455
2026-09-28T21:53:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2708 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:53:42 dist-test-slave-jrck krb5kdc[25678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:53:42 dist-test-slave-jrck krb5kdc[25678](info): set up 2 sockets
Sep 28 21:53:42 dist-test-slave-jrck krb5kdc[25678](info): commencing operation
krb5kdc: starting...
W20260928 21:53:43.740439 3075 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/test-user.keytab.
Sep 28 21:53:43 dist-test-slave-jrck krb5kdc[25678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.0.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
Entry for principal kudu/127.3.0.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab.
I20260928 21:53:43.855041 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:44189
--webserver_interface=127.3.0.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:44189,127.3.0.253:37609,127.3.0.252:37701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:53:43.934839 25691 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:53:43.935078 25691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:43.935096 25691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:43.935117 25691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:43.936270 25691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:43.936306 25691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:43.936319 25691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:43.936331 25691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:43.936342 25691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:43.937916 25691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:44189,127.3.0.253:37609,127.3.0.252:37701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.254.keytab
--principal=kudu/127.3.0.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.254:44189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.0.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:43.938143 25691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:43.938328 25691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:43.942996 25696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:43.943017 25697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:43.943149 25691 server_base.cc:1034] running on GCE node
W20260928 21:53:43.942988 25699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:43.943437 25691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:43.943743 25691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:43.944898 25691 hybrid_clock.cc:648] HybridClock initialized: now 1790632423944882 us; error 30 us; skew 500 ppm
Sep 28 21:53:43 dist-test-slave-jrck krb5kdc[25678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632423, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:43.947515 25691 init.cc:377] Logged in from keytab as kudu/127.3.0.254@KRBTEST.COM (short username kudu)
I20260928 21:53:43.947973 25691 webserver.cc:460] Webserver started at http://127.3.0.254:43689/ using document root <none> and password file <none>
I20260928 21:53:43.948192 25691 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:43.948242 25691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:43.948371 25691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:43.949506 25691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/data/instance:
uuid: "cd21634eecb1442a8de283dcbe4f9326"
format_stamp: "Formatted at 2026-09-28 21:53:43 on dist-test-slave-jrck"
server_key: "751f4c3dcd76041699e4b843a96924b1"
server_key_iv: "039e9d27095b91c03a9cd378cbc628e0"
server_key_version: "encryptionkey@0"
I20260928 21:53:43.949837 25691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/wal/instance:
uuid: "cd21634eecb1442a8de283dcbe4f9326"
format_stamp: "Formatted at 2026-09-28 21:53:43 on dist-test-slave-jrck"
server_key: "751f4c3dcd76041699e4b843a96924b1"
server_key_iv: "039e9d27095b91c03a9cd378cbc628e0"
server_key_version: "encryptionkey@0"
I20260928 21:53:43.951145 25691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:53:43.951869 25706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:43.952078 25691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:43.952176 25691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/wal
uuid: "cd21634eecb1442a8de283dcbe4f9326"
format_stamp: "Formatted at 2026-09-28 21:53:43 on dist-test-slave-jrck"
server_key: "751f4c3dcd76041699e4b843a96924b1"
server_key_iv: "039e9d27095b91c03a9cd378cbc628e0"
server_key_version: "encryptionkey@0"
I20260928 21:53:43.952241 25691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:43.962788 25691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:43.964768 25691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:43.964926 25691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:43.968837 25691 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.254:44189
I20260928 21:53:43.968864 25758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.254:44189 every 8 connection(s)
I20260928 21:53:43.969264 25691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/data/info.pb
I20260928 21:53:43.969805 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 25691
I20260928 21:53:43.969885 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-0/wal/instance
I20260928 21:53:43.970070 3075 external_mini_cluster.cc:1428] Setting key 5f356617e75c2e3cb3ce926983430e9b
I20260928 21:53:43.970258 25759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:43.971817 25759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.0.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.253@KRBTEST.COM" created.
Sep 28 21:53:43 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632423, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:43.983528 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:43.972564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.3.0.254:44189)
0928 21:53:43.972733 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:43.972743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:43.972852 (+ 109us) server_negotiation.cc:407] Connection header received
0928 21:53:43.973439 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:43.973441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:43.973458 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:43.973540 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:43.974070 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:43.974887 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:43.975720 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:43.976029 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:43.976139 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:43.980269 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:43.980279 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:43.980280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:43.980289 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:43.981865 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:43.982294 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:43.982297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:43.982297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:43.982370 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:43.982608 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:43.982609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:43.982609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:43.982902 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:43.982977 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:43.983230 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 21:53:43.983333 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"spinlock_wait_cycles":8064,"thread_start_us":88,"threads_started":1}
I20260928 21:53:43.983541 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:43.972547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44189 (local address 127.0.0.1:51980)
0928 21:53:43.972724 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:43.972763 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:43.973290 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:43.973551 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:43.973553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:43.973822 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:43.973975 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:43.973979 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:43.974970 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:43.974972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:43.975643 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:43.975646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:43.976129 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:43.976130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:43.976246 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:43.976801 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:43.976810 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:43.980166 (+ 3356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:43.981951 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:43.981959 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:43.981962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:43.982224 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:43.982419 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:43.982420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:43.982420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:43.982527 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:43.982975 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:43.982983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:43.983072 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 21:53:43.983178 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 21:53:43.983424 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":73,"thread_start_us":87,"threads_started":1}
I20260928 21:53:43.984304 25759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:43.984686 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:43.984567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:37609 (local address 127.0.0.1:32914)
0928 21:53:43.984603 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:43.984659 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:37609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 21:53:43.984834 25708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.253:37609: connect: Connection refused (error 111)
I20260928 21:53:43.985265 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:43.985165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:37609 (local address 127.0.0.1:32916)
0928 21:53:43.985211 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:43.985242 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:37609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 21:53:43.985366 25759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.253:37609: Network error: Client connection negotiation failed: client connection to 127.3.0.253:37609: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab.
Entry for principal kudu/127.3.0.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab.
I20260928 21:53:43.994771 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.253:37609
--webserver_interface=127.3.0.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab
--principal=kudu/127.3.0.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:44189,127.3.0.253:37609,127.3.0.252:37701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:53:44.034612 25759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } attempt: 1
I20260928 21:53:44.035192 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.035038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:37609 (local address 127.0.0.1:32926)
0928 21:53:44.035101 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.035155 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:37609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 21:53:44.035553 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.035466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:37609 (local address 127.0.0.1:32940)
0928 21:53:44.035506 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.035534 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:37609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:53:44.035646 25759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.253:37609: Network error: Client connection negotiation failed: client connection to 127.3.0.253:37609: connect: Connection refused (error 111)
W20260928 21:53:44.079123 25766 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:53:44.079335 25766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:44.079352 25766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:44.079372 25766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:44.080582 25766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:44.080624 25766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:44.080637 25766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:44.080649 25766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:44.080665 25766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:44.082194 25766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:44189,127.3.0.253:37609,127.3.0.252:37701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.253.keytab
--principal=kudu/127.3.0.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.253:37609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.0.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:44.082399 25766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:44.082587 25766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:44.087123 25771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:44.087136 25772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:44.087138 25774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:44.087402 25766 server_base.cc:1034] running on GCE node
I20260928 21:53:44.087689 25766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:44.087963 25766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:44.089102 25766 hybrid_clock.cc:648] HybridClock initialized: now 1790632424089095 us; error 32 us; skew 500 ppm
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:44.091406 25766 init.cc:377] Logged in from keytab as kudu/127.3.0.253@KRBTEST.COM (short username kudu)
I20260928 21:53:44.091794 25766 webserver.cc:460] Webserver started at http://127.3.0.253:35589/ using document root <none> and password file <none>
I20260928 21:53:44.091965 25766 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:44.092012 25766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:44.092101 25766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:44.093154 25766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/data/instance:
uuid: "8669a5c20f3b4688887f43ed9ace1383"
format_stamp: "Formatted at 2026-09-28 21:53:44 on dist-test-slave-jrck"
server_key: "ca38ca25b4ad44a6ce5359353a686bd7"
server_key_iv: "e7d0ba3f77190e675756feed259deb60"
server_key_version: "encryptionkey@0"
I20260928 21:53:44.093447 25766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/wal/instance:
uuid: "8669a5c20f3b4688887f43ed9ace1383"
format_stamp: "Formatted at 2026-09-28 21:53:44 on dist-test-slave-jrck"
server_key: "ca38ca25b4ad44a6ce5359353a686bd7"
server_key_iv: "e7d0ba3f77190e675756feed259deb60"
server_key_version: "encryptionkey@0"
I20260928 21:53:44.094601 25766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:53:44.095252 25781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:44.095415 25766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 21:53:44.095489 25766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/wal
uuid: "8669a5c20f3b4688887f43ed9ace1383"
format_stamp: "Formatted at 2026-09-28 21:53:44 on dist-test-slave-jrck"
server_key: "ca38ca25b4ad44a6ce5359353a686bd7"
server_key_iv: "e7d0ba3f77190e675756feed259deb60"
server_key_version: "encryptionkey@0"
I20260928 21:53:44.095552 25766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:44.103780 25759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } attempt: 2
I20260928 21:53:44.104190 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.104077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:37609 (local address 127.0.0.1:32950)
0928 21:53:44.104126 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.104163 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:37609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 21:53:44.104534 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.104395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:37609 (local address 127.0.0.1:32954)
0928 21:53:44.104479 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.104512 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.253:37609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 21:53:44.104611 25759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.253:37609: Network error: Client connection negotiation failed: client connection to 127.3.0.253:37609: connect: Connection refused (error 111)
I20260928 21:53:44.140201 25766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:44.142457 25766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:44.142599 25766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:53:44.146318 25766 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.253:37609
I20260928 21:53:44.146344 25833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.253:37609 every 8 connection(s)
I20260928 21:53:44.146689 25766 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/data/info.pb
I20260928 21:53:44.147672 25834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:44.149310 25834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:44.151814 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 25766
I20260928 21:53:44.151909 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-1/wal/instance
I20260928 21:53:44.152094 3075 external_mini_cluster.cc:1428] Setting key e012e00f9e876e8ce479731f104241fd
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:44.158380 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.150051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.3.0.254:44189)
0928 21:53:44.150112 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.150113 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.150309 (+ 196us) server_negotiation.cc:407] Connection header received
0928 21:53:44.150769 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.150770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.150782 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.150844 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.151450 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.152128 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.153163 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.153378 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.153435 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.155647 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.155648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.155649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.155657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.157077 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.157451 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.157452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.157452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.157519 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.157731 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.157732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.157733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.157851 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.157906 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.158200 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.158287 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:53:44.158529 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.150011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44189 (local address 127.0.0.1:51986)
0928 21:53:44.150190 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.150233 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:44.150694 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:44.150908 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:44.150911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:44.151175 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:44.151372 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.151376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.152223 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.152224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.153079 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.153082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.153465 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.153466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.153573 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.154056 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:44.154064 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:44.155543 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:44.157156 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.157164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.157166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.157385 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.157570 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.157571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.157572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.157679 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.157902 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:44.157909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:44.158060 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 21:53:44.158151 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:53:44.158312 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.3.0.252@KRBTEST.COM; defaulting to no policy
I20260928 21:53:44.159058 25834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.0.252@KRBTEST.COM" created.
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
I20260928 21:53:44.166515 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.159290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:37609 (local address 127.0.0.1:32962)
0928 21:53:44.159336 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.159348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:44.159386 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:44.159628 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:44.159629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:44.159768 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:44.159824 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.159826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.160848 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.160849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.161457 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.161459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.161916 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.161917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.161979 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.162402 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:44.162412 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:44.163737 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:44.165542 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.165543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.165543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.165703 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.165879 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.165880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.165880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.165934 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.166190 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:44.166191 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:44.166231 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:53:44.166292 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:53:44.166412 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 21:53:44.166606 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.159321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.3.0.253:37609)
0928 21:53:44.159461 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.159466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.159494 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:53:44.159513 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.159514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.159524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.159595 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.159928 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.160765 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.161538 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.161853 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.161912 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.163822 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.163833 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.163833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.163841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.165460 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.165773 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.165776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.165777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.165826 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.165982 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.165983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.165984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.166133 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.166202 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.166331 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.166493 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260928 21:53:44.167251 25834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:44.167600 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.167476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36398)
0928 21:53:44.167520 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.167571 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 21:53:44.167737 25784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
I20260928 21:53:44.168118 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.168033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36408)
0928 21:53:44.168069 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.168093 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 21:53:44.168246 25834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:37701: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.0.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab.
Entry for principal kudu/127.3.0.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab.
I20260928 21:53:44.177474 3075 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksdiuc7/build/release/bin/kudu
/tmp/dist-test-tasksdiuc7/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.252:37701
--webserver_interface=127.3.0.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab
--principal=kudu/127.3.0.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.0.212:39885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.0.254:44189,127.3.0.253:37609,127.3.0.252:37701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:53:44.195766 25759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } attempt: 3
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632423, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
I20260928 21:53:44.202639 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.196237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.3.0.253:37609)
0928 21:53:44.196302 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.196304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.196325 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:53:44.196353 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.196354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.196367 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.196408 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.196757 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.197258 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.197988 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.198229 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.198289 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.200200 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.200202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.200203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.200211 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.201509 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.201875 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.201877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.201877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.201941 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.202111 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.202111 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.202112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.202229 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.202286 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.202439 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.202532 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:53:44.202661 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.196172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:37609 (local address 127.0.0.1:32978)
0928 21:53:44.196222 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.196235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:44.196285 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:44.196428 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:44.196429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:44.196607 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:44.196676 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.196679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.197344 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.197346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.197902 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.197904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.198308 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.198308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.198381 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.198869 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:44.198874 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:44.200100 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:44.201593 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.201594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.201594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.201810 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.201998 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.201999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.202000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.202060 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.202294 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:44.202295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:44.202348 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 21:53:44.202415 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:53:44.202533 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 21:53:44.203130 25759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:44.203482 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.203383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36414)
0928 21:53:44.203432 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.203462 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:53:44.203780 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.203693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36420)
0928 21:53:44.203733 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.203756 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 21:53:44.203856 25759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:37701: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
I20260928 21:53:44.217468 25834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } attempt: 1
I20260928 21:53:44.217924 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.217800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36426)
0928 21:53:44.217855 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.217889 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:53:44.218233 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.218154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36434)
0928 21:53:44.218190 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.218214 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 21:53:44.218338 25834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:37701: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
I20260928 21:53:44.234982 25759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } attempt: 1
I20260928 21:53:44.235491 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.235322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36438)
0928 21:53:44.235403 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.235459 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:53:44.235888 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.235801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36440)
0928 21:53:44.235842 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.235869 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 21:53:44.236001 25759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:37701: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
W20260928 21:53:44.259778 25841 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:53:44.260007 25841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:53:44.260033 25841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:53:44.260061 25841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:53:44.261245 25841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:53:44.261288 25841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:53:44.261307 25841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:53:44.261322 25841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:53:44.261337 25841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:53:44.262892 25841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.0.212:39885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.0.254:44189,127.3.0.253:37609,127.3.0.252:37701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/krb5kdc/kudu_127.3.0.252.keytab
--principal=kudu/127.3.0.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.0.252:37701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.0.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 20:39:55 UTC on ccycloud.quasar-gjkqpn.root.comops.site
build id 4678653
I20260928 21:53:44.263147 25841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:53:44.263361 25841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:53:44.267941 25847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:44.267982 25849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:53:44.267992 25846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:53:44.268215 25841 server_base.cc:1034] running on GCE node
I20260928 21:53:44.268448 25841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:53:44.268774 25841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:53:44.269963 25841 hybrid_clock.cc:648] HybridClock initialized: now 1790632424269939 us; error 33 us; skew 500 ppm
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:53:44.272445 25841 init.cc:377] Logged in from keytab as kudu/127.3.0.252@KRBTEST.COM (short username kudu)
I20260928 21:53:44.272950 25841 webserver.cc:460] Webserver started at http://127.3.0.252:37647/ using document root <none> and password file <none>
I20260928 21:53:44.273149 25841 fs_manager.cc:359] Metadata directory not provided
I20260928 21:53:44.273196 25841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:53:44.273317 25841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:53:44.274385 25841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/data/instance:
uuid: "fdac15ce97d04dca80a731830a89b162"
format_stamp: "Formatted at 2026-09-28 21:53:44 on dist-test-slave-jrck"
server_key: "de5f1c5106796b3f5c14296123c75e51"
server_key_iv: "68a0a9b67d605ceffb3115214ba1fa71"
server_key_version: "encryptionkey@0"
I20260928 21:53:44.274718 25841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/wal/instance:
uuid: "fdac15ce97d04dca80a731830a89b162"
format_stamp: "Formatted at 2026-09-28 21:53:44 on dist-test-slave-jrck"
server_key: "de5f1c5106796b3f5c14296123c75e51"
server_key_iv: "68a0a9b67d605ceffb3115214ba1fa71"
server_key_version: "encryptionkey@0"
I20260928 21:53:44.275975 25841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 21:53:44.276757 25856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:53:44.276916 25841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:53:44.277009 25841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/data,/tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/wal
uuid: "fdac15ce97d04dca80a731830a89b162"
format_stamp: "Formatted at 2026-09-28 21:53:44 on dist-test-slave-jrck"
server_key: "de5f1c5106796b3f5c14296123c75e51"
server_key_iv: "68a0a9b67d605ceffb3115214ba1fa71"
server_key_version: "encryptionkey@0"
I20260928 21:53:44.277093 25841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:53:44.285000 25841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:53:44.286500 25834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } attempt: 2
I20260928 21:53:44.286873 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.286739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36450)
0928 21:53:44.286797 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.286838 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:53:44.287212 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.287118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36452)
0928 21:53:44.287163 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.287191 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 21:53:44.287204 25841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:53:44.287340 25841 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260928 21:53:44.287416 25834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.0.252:37701: Network error: Client connection negotiation failed: client connection to 127.3.0.252:37701: connect: Connection refused (error 111)
I20260928 21:53:44.291051 25841 rpc_server.cc:307] RPC server started. Bound to: 127.3.0.252:37701
I20260928 21:53:44.291085 25908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.0.252:37701 every 8 connection(s)
I20260928 21:53:44.291390 25841 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/data/info.pb
I20260928 21:53:44.292189 3075 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksdiuc7/build/release/bin/kudu as pid 25841
I20260928 21:53:44.292294 3075 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksdiuc7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790632139349757-3075-0/minicluster-data/master-2/wal/instance
I20260928 21:53:44.292481 25909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:53:44.292548 3075 external_mini_cluster.cc:1428] Setting key f475367b2c534115763e034b09ed747b
I20260928 21:53:44.294700 25909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:44.302539 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.295380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.3.0.254:44189)
0928 21:53:44.295429 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.295431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.295649 (+ 218us) server_negotiation.cc:407] Connection header received
0928 21:53:44.295677 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.295678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.295688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.295729 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.296253 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.296763 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.297761 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.297982 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.298051 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.300266 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.300267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.300268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.300274 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.301468 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.301795 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.301796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.301796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.301843 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.302022 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.302023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.302023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.302139 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.302199 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.302363 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.302447 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:53:44.302717 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.293036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.3.0.252:37701)
0928 21:53:44.293209 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.293218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.293249 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:53:44.293976 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.293987 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.294054 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.294216 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.294785 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.296052 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.296859 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.297444 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.297561 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.299775 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.299782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.299782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.299788 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.301442 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.301756 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.301759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.301759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.301842 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.302030 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.302030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.302031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.302251 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.302380 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.302468 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.302553 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260928 21:53:44.302978 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.295316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.254:44189 (local address 127.0.0.1:51988)
0928 21:53:44.295495 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.295536 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:44.295609 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:44.295758 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:44.295760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:44.296103 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:44.296179 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.296181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.296842 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.296843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.297681 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.297683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.298079 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.298080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.298132 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.298559 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:44.298568 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:44.300188 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:44.301545 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.301551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.301553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.301723 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.301893 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.301894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.301894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.301967 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.302197 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:44.302201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:44.302255 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 21:53:44.302341 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:53:44.302489 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260928 21:53:44.303464 25909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.254@KRBTEST.COM
I20260928 21:53:44.310411 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.303695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.253:37609 (local address 127.0.0.1:32986)
0928 21:53:44.303735 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.303744 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:44.303782 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:44.303914 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:44.303914 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:44.304053 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:44.304094 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.304095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.304929 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.304929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.305671 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.305673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.306143 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.306144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.306186 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.306514 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:44.306518 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:44.307736 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:44.309259 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.309260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.309260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.309421 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.309632 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.309633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.309633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.309681 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.310139 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:44.310140 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:44.310175 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:53:44.310228 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:53:44.310317 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 21:53:44.310441 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.303768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.3.0.253:37609)
0928 21:53:44.303812 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.303814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.303821 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:53:44.303843 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.303843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.303852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.303895 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.304167 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.304863 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.305749 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.306074 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.306140 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.307817 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.307818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.307819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.307825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.309184 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.309504 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.309505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.309505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.309570 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.309732 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.309733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.309734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.309867 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.309932 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.310279 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.310351 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:53:44.310802 25909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:53:44.310999 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.303507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52002 (local address 127.3.0.254:44189)
0928 21:53:44.303551 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.303553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.303562 (+ 9us) server_negotiation.cc:407] Connection header received
0928 21:53:44.303735 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.303736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.303746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.303797 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.304314 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.305001 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.305898 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.306157 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.306211 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.308548 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.308549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.308549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.308556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.309871 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.310127 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.310128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.310128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.310172 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.310357 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.310358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.310358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.310460 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.310620 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.310834 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.310914 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:53:44.311116 25759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } attempt: 2
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.252@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632423, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.254@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.0.253@KRBTEST.COM
I20260928 21:53:44.317965 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.311158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.3.0.252:37701)
0928 21:53:44.311411 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.311413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.311420 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:53:44.311437 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.311437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.311446 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.311502 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.311745 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.312481 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.313080 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.313362 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.313425 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.315197 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.315198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.315199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.315205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.316836 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.317237 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.317238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.317238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.317289 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.317446 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.317447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.317448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.317599 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.317652 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.317778 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.317862 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234}
I20260928 21:53:44.318223 25913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.311075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36460)
0928 21:53:44.311214 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.311225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:44.311266 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:44.311511 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:44.311512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:44.311652 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:44.311686 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.311688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.312536 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.312537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.313014 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.313015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.313426 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.313426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.313468 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.313936 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:44.313941 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:44.315130 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:44.316920 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.316920 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.316921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.317183 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.317344 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.317345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.317345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.317398 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.317655 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:44.317656 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:44.317694 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:53:44.317753 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:53:44.317909 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":71}
I20260928 21:53:44.318734 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.311281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36472)
0928 21:53:44.311318 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.311327 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:44.311370 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:44.311703 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:44.311704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:44.311841 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:44.311877 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.311879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.313193 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.313194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.313763 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.313765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.314192 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.314193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.314252 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.314630 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:44.314635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:44.315910 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:44.317702 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.317703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.317704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.317903 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.318131 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.318131 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.318132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.318188 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.318465 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:44.318467 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:44.318513 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 21:53:44.318560 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:53:44.318649 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 21:53:44.319061 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.311361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.3.0.252:37701)
0928 21:53:44.311585 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.311586 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.311592 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:53:44.311605 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.311606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.311614 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.311936 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.311969 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.313110 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.313833 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.314108 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.314402 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.315987 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.315989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.315989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.315996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.317620 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.317985 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.317986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.317987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.318057 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.318255 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.318255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.318256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.318376 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.318848 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.318888 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.318959 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":114,"threads_started":1}
I20260928 21:53:44.319825 25909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162: Bootstrap starting.
I20260928 21:53:44.320618 25909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:44.320618 25759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326: Bootstrap starting.
I20260928 21:53:44.320945 25909 log.cc:824] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:44.321276 25759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:44.321570 25759 log.cc:824] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:44.321781 25909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162: No bootstrap required, opened a new log
I20260928 21:53:44.322296 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.311646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.3.0.253:37609)
0928 21:53:44.311679 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.311680 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.311687 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:53:44.311779 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.311780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.311791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.311840 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.312366 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.313113 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.314681 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.314976 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.315018 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.317553 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.317555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.317556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.317563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.320984 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.321444 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.321445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.321446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.321506 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.321673 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.321674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.321674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.321784 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.321937 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.322137 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.322211 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 21:53:44.322387 25759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326: No bootstrap required, opened a new log
I20260928 21:53:44.323525 3075 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:53:44.323668 25909 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } }
I20260928 21:53:44.323818 25909 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdac15ce97d04dca80a731830a89b162, State: Initialized, Role: FOLLOWER
I20260928 21:53:44.323894 25909 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } }
I20260928 21:53:44.324229 25909 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:44.324225 25917 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } } }
I20260928 21:53:44.324271 25733 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cd21634eecb1442a8de283dcbe4f9326"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52002
I20260928 21:53:44.324359 25917 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:44.324298 25759 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } }
I20260928 21:53:44.324417 25759 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd21634eecb1442a8de283dcbe4f9326, State: Initialized, Role: FOLLOWER
I20260928 21:53:44.324529 25759 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } }
I20260928 21:53:44.324849 25759 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:44.324815 25918 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } } }
I20260928 21:53:44.324872 25733 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 21:53:44.324889 25918 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:44.324921 25733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:44.325655 25733 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } }
I20260928 21:53:44.325897 25733 leader_election.cc:288] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [CANDIDATE]: Term 1 election: Requested vote from peers 8669a5c20f3b4688887f43ed9ace1383 (127.3.0.253:37609), fdac15ce97d04dca80a731830a89b162 (127.3.0.252:37701)
I20260928 21:53:44.326385 25883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd21634eecb1442a8de283dcbe4f9326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fdac15ce97d04dca80a731830a89b162"
I20260928 21:53:44.326504 25883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:53:44.327307 25883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd21634eecb1442a8de283dcbe4f9326 in term 1.
I20260928 21:53:44.327553 25709 leader_election.cc:302] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd21634eecb1442a8de283dcbe4f9326, fdac15ce97d04dca80a731830a89b162; no voters:
I20260928 21:53:44.327667 25918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:53:44.327783 25918 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [term 1 LEADER]: Becoming Leader. State: Replica: cd21634eecb1442a8de283dcbe4f9326, State: Running, Role: LEADER
I20260928 21:53:44.327857 25918 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } }
I20260928 21:53:44.328248 25918 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd21634eecb1442a8de283dcbe4f9326. Latest consensus state: current_term: 1 leader_uuid: "cd21634eecb1442a8de283dcbe4f9326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } } }
I20260928 21:53:44.328323 25918 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:44.328801 25947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 21:53:44.329275 25942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:53:44.329336 25942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:53:44.329408 25947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:53:44.327921 25691 master_runner.cc:186] Error getting master registration for 127.3.0.253:37609: OK, instance_id { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" instance_seqno: 1790632424145258 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:53:44.330770 25691 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:53:44.330739 25841 master_runner.cc:186] Error getting master registration for 127.3.0.253:37609: OK, instance_id { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" instance_seqno: 1790632424145258 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:53:44.332924 25841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:53:44.336246 25883 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [term 1 FOLLOWER]: Refusing update from remote peer cd21634eecb1442a8de283dcbe4f9326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:44.337512 25940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:44.338945 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.328351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.3.0.252:37701)
0928 21:53:44.328399 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.328401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.330348 (+ 1947us) server_negotiation.cc:407] Connection header received
0928 21:53:44.330441 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.330443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.330454 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.330483 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.331748 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.332506 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.333781 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.334071 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.334140 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.335579 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.335580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.335580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.335587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.337200 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.338083 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.338084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.338085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.338145 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.338344 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.338346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.338346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.338477 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.338556 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.338761 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.338845 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 21:53:44.339417 25917 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd21634eecb1442a8de283dcbe4f9326. Latest consensus state: current_term: 1 leader_uuid: "cd21634eecb1442a8de283dcbe4f9326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } } }
I20260928 21:53:44.339516 25917 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:44.340564 25918 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd21634eecb1442a8de283dcbe4f9326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } } }
I20260928 21:53:44.340656 25918 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [sys.catalog]: This master's current role is: LEADER
I20260928 21:53:44.341035 25917 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd21634eecb1442a8de283dcbe4f9326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } } }
I20260928 21:53:44.341113 25917 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:44.341415 25947 catalog_manager.cc:1347] Generated new cluster ID: a5e03acc16424a44bb87a58a936c31e4
I20260928 21:53:44.341459 25947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:53:44.349067 25763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.328544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.3.0.254:44189)
0928 21:53:44.328593 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.328595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.329490 (+ 895us) server_negotiation.cc:407] Connection header received
0928 21:53:44.329587 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.329588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.329598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.329623 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.330368 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.331060 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.334848 (+ 3788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.335155 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.335198 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.337311 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.337313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.337313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.337320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.338891 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.341141 (+ 2250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.341142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.341142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.341204 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.348228 (+ 7024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.348230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.348230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.348404 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.348555 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.348876 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.348969 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:53:44.351348 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.329920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.3.0.253:37609)
0928 21:53:44.329971 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.329973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.332132 (+ 2159us) server_negotiation.cc:407] Connection header received
0928 21:53:44.332220 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.332221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.332230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.332256 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.333133 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.333824 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.334812 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.335128 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.335197 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.348246 (+ 13049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.348248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.348248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.348258 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.350088 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.350497 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.350499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.350499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.350580 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.350783 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.350783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.350784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.350918 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.351004 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.351170 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.351249 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:53:44.351714 25954 mvcc.cc:205] Tried to move back new op lower bound from 7334430410072260608 to 7334430410047627264. Current Snapshot: MvccSnapshot[applied={T|T < 7334430410072260608}]
I20260928 21:53:44.352594 25953 mvcc.cc:205] Tried to move back new op lower bound from 7334430410072260608 to 7334430410047627264. Current Snapshot: MvccSnapshot[applied={T|T < 7334430410072260608 or (T in {7334430410072260608})}]
I20260928 21:53:44.366899 25947 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:53:44.367736 25947 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:53:44.373555 25947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326: Generated new TSK 0
I20260928 21:53:44.373805 25947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:53:44.378501 25834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } attempt: 3
Sep 28 21:53:44 dist-test-slave-jrck krb5kdc[25678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790632424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.0.253@KRBTEST.COM for kudu/127.3.0.252@KRBTEST.COM
I20260928 21:53:44.386282 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.378793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.0.252:37701 (local address 127.0.0.1:36486)
0928 21:53:44.378872 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 21:53:44.378883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:53:44.378920 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:53:44.379128 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:53:44.379129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:53:44.379315 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:53:44.379366 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.379368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.380327 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.380328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.381111 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:53:44.381115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.381604 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.381606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:53:44.381664 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.382042 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:53:44.382048 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:53:44.383569 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:53:44.385279 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.385280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.385281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.385489 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.385653 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:53:44.385654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:53:44.385654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:53:44.385724 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:53:44.385972 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:53:44.385973 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:53:44.386021 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 21:53:44.386082 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:53:44.386181 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:53:44.386281 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:44.378914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.3.0.252:37701)
0928 21:53:44.378999 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.379001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.379007 (+ 6us) server_negotiation.cc:407] Connection header received
0928 21:53:44.379028 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.379028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.379037 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.379077 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.379467 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.380140 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.381203 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:44.381475 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:44.381535 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:44.383646 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:53:44.383648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:53:44.383648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:53:44.383656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:53:44.385198 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.385542 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.385543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.385544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.385597 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:53:44.385769 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:53:44.385770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:53:44.385771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:53:44.385912 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:53:44.385974 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:44.386120 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 21:53:44.386191 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:53:44.388306 25834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383: Bootstrap starting.
I20260928 21:53:44.389065 25834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383: Neither blocks nor log segments found. Creating new log.
I20260928 21:53:44.389349 25834 log.cc:824] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383: Log is configured to *not* fsync() on all Append() calls
I20260928 21:53:44.390244 25834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383: No bootstrap required, opened a new log
I20260928 21:53:44.392203 25834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } }
I20260928 21:53:44.392351 25834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8669a5c20f3b4688887f43ed9ace1383, State: Initialized, Role: FOLLOWER
I20260928 21:53:44.392436 25834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } }
I20260928 21:53:44.392794 25834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:53:44.392850 25808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd21634eecb1442a8de283dcbe4f9326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8669a5c20f3b4688887f43ed9ace1383"
I20260928 21:53:44.392951 25808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [term 0 FOLLOWER]: Advancing to term 1
W20260928 21:53:44.393191 25787 tablet.cc:2369] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:53:44.393945 25808 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd21634eecb1442a8de283dcbe4f9326 in term 1.
I20260928 21:53:44.394421 25807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [term 1 FOLLOWER]: Refusing update from remote peer cd21634eecb1442a8de283dcbe4f9326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:53:44.394896 25940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:53:44.392784 25960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } } }
I20260928 21:53:44.395946 25960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:44.397200 25960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd21634eecb1442a8de283dcbe4f9326. Latest consensus state: current_term: 1 leader_uuid: "cd21634eecb1442a8de283dcbe4f9326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } } }
I20260928 21:53:44.397300 25960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:53:44.397347 25960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd21634eecb1442a8de283dcbe4f9326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } } }
I20260928 21:53:44.397397 25960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:53:44.398003 25971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:53:44.398079 25971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:53:44.398581 25973 mvcc.cc:205] Tried to move back new op lower bound from 7334430410229399552 to 7334430410047627264. Current Snapshot: MvccSnapshot[applied={T|T < 7334430410072260608}]
W20260928 21:53:44.407083 25914 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.405370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.3.0.252:37701)
0928 21:53:44.405416 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.405418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.406390 (+ 972us) server_negotiation.cc:407] Connection header received
0928 21:53:44.406496 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.406497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.406508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.406539 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.407009 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36502: BlockingRecv error: recv got EOF from 127.0.0.1:36502 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 21:53:44.407913 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.405768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.3.0.253:37609)
0928 21:53:44.405805 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.405807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.407418 (+ 1611us) server_negotiation.cc:407] Connection header received
0928 21:53:44.407513 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.407514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.407524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.407553 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.407850 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33002: BlockingRecv error: recv got EOF from 127.0.0.1:33002 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 21:53:44.408766 25763 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.406117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.3.0.254:44189)
0928 21:53:44.406157 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.406160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.408221 (+ 2061us) server_negotiation.cc:407] Connection header received
0928 21:53:44.408318 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.408319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.408328 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.408356 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.408705 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52024: BlockingRecv error: recv got EOF from 127.0.0.1:52024 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 21:53:44.417753 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.416807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.3.0.253:37609)
0928 21:53:44.416855 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.416857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.417142 (+ 285us) server_negotiation.cc:407] Connection header received
0928 21:53:44.417254 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.417255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.417275 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.417306 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.417692 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33008: BlockingRecv error: recv got EOF from 127.0.0.1:33008 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 21:53:44.421169 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.420216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.3.0.253:37609)
0928 21:53:44.420255 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.420257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.420563 (+ 306us) server_negotiation.cc:407] Connection header received
0928 21:53:44.420660 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.420662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.420673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.420701 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.421062 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33012: BlockingRecv error: recv got EOF from 127.0.0.1:33012 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 21:53:44.425338 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.424485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.3.0.253:37609)
0928 21:53:44.424522 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.424523 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.424751 (+ 228us) server_negotiation.cc:407] Connection header received
0928 21:53:44.424862 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.424863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.424875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.424905 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.425283 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33016: BlockingRecv error: recv got EOF from 127.0.0.1:33016 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 21:53:44.431495 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.430654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.3.0.253:37609)
0928 21:53:44.430692 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.430694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.430940 (+ 246us) server_negotiation.cc:407] Connection header received
0928 21:53:44.431038 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.431040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.431053 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.431084 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.431437 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33022: BlockingRecv error: recv got EOF from 127.0.0.1:33022 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 21:53:44.441805 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.440927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.3.0.253:37609)
0928 21:53:44.440970 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.440972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.441231 (+ 259us) server_negotiation.cc:407] Connection header received
0928 21:53:44.441328 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.441330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.441342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.441371 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.441731 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33024: BlockingRecv error: recv got EOF from 127.0.0.1:33024 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 21:53:44.460145 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.459249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.3.0.253:37609)
0928 21:53:44.459291 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.459293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.459561 (+ 268us) server_negotiation.cc:407] Connection header received
0928 21:53:44.459655 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.459656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.459668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.459696 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.460080 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33026: BlockingRecv error: recv got EOF from 127.0.0.1:33026 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 21:53:44.494601 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.493647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.3.0.253:37609)
0928 21:53:44.493692 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.493694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.493990 (+ 296us) server_negotiation.cc:407] Connection header received
0928 21:53:44.494093 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.494095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.494108 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.494136 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.494538 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33030: BlockingRecv error: recv got EOF from 127.0.0.1:33030 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 21:53:44.560822 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.560135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.3.0.253:37609)
0928 21:53:44.560223 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.560224 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.560235 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:53:44.560322 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.560323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.560334 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.560384 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.560754 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33044: BlockingRecv error: recv got EOF from 127.0.0.1:33044 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 21:53:44.690488 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.689890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.3.0.253:37609)
0928 21:53:44.689969 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.689971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.690034 (+ 63us) server_negotiation.cc:407] Connection header received
0928 21:53:44.690076 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.690077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.690088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.690144 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.690428 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33060: BlockingRecv error: recv got EOF from 127.0.0.1:33060 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:53:44.863718 25940 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd21634eecb1442a8de283dcbe4f9326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd21634eecb1442a8de283dcbe4f9326" member_type: VOTER last_known_addr { host: "127.3.0.254" port: 44189 } } peers { permanent_uuid: "8669a5c20f3b4688887f43ed9ace1383" member_type: VOTER last_known_addr { host: "127.3.0.253" port: 37609 } } peers { permanent_uuid: "fdac15ce97d04dca80a731830a89b162" member_type: VOTER last_known_addr { host: "127.3.0.252" port: 37701 } } }
I20260928 21:53:44.863862 25940 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd21634eecb1442a8de283dcbe4f9326 [sys.catalog]: This master's current role is: LEADER
W20260928 21:53:44.948333 25838 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:53:44.947756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.3.0.253:37609)
0928 21:53:44.947814 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:44.947816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:44.947827 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:53:44.947888 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:44.947888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:44.947897 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:44.947947 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:53:44.948239 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33066: BlockingRecv error: recv got EOF from 127.0.0.1:33066 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:53:45.329771 25942 catalog_manager.cc:1259] Loaded cluster ID: a5e03acc16424a44bb87a58a936c31e4
I20260928 21:53:45.329854 25942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162: loading cluster ID for follower catalog manager: success
I20260928 21:53:45.330677 25942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162: acquiring CA information for follower catalog manager: success
I20260928 21:53:45.330989 25942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fdac15ce97d04dca80a731830a89b162: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:53:45.398571 25971 catalog_manager.cc:1259] Loaded cluster ID: a5e03acc16424a44bb87a58a936c31e4
I20260928 21:53:45.398651 25971 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383: loading cluster ID for follower catalog manager: success
I20260928 21:53:45.399569 25971 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383: acquiring CA information for follower catalog manager: success
I20260928 21:53:45.399863 25971 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8669a5c20f3b4688887f43ed9ace1383: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:53:45.464035 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:45.461494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.3.0.253:37609)
0928 21:53:45.461621 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:45.461623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:45.461633 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:53:45.461679 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:45.461680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:45.461735 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:45.461823 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:45.461976 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:45.462467 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:45.463237 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:45.463453 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:45.463511 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:45.463680 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:45.463815 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:45.463886 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:45.463923 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 21:53:45.463946 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
I20260928 21:53:45.467952 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:45.465687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.3.0.252:37701)
0928 21:53:45.465820 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:45.465822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:45.465829 (+ 7us) server_negotiation.cc:407] Connection header received
0928 21:53:45.465863 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:45.465864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:45.465898 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:45.465984 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:45.466106 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:45.466594 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:45.467365 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:45.467543 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:45.467583 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:45.467683 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:45.467780 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:45.467829 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:45.467857 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 21:53:45.467872 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260928 21:53:45.472182 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:45.469427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.3.0.252:37701)
0928 21:53:45.469535 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:45.469537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:45.469791 (+ 254us) server_negotiation.cc:407] Connection header received
0928 21:53:45.469886 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:45.469887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:45.469898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:45.469924 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:45.470090 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:45.470750 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:45.471459 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:45.471732 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:45.471790 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:45.471941 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:45.471975 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:45.472018 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:45.472095 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 21:53:45.472116 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 21:53:45.472226 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:45.469519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.3.0.253:37609)
0928 21:53:45.469552 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:45.469554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:45.469649 (+ 95us) server_negotiation.cc:407] Connection header received
0928 21:53:45.469701 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:45.469701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:45.469712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:45.469762 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:45.469901 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:45.470611 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:45.471592 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:45.471818 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:45.471879 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:45.471993 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:45.472022 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:45.472064 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:45.472146 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 21:53:45.472168 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 21:53:45.472553 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:53:45.469537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.3.0.254:44189)
0928 21:53:45.469635 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 21:53:45.469636 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 21:53:45.469644 (+ 8us) server_negotiation.cc:407] Connection header received
0928 21:53:45.469725 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:53:45.469725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:53:45.469775 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:53:45.469847 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:53:45.469968 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:45.470504 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:45.471551 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:53:45.471849 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:53:45.471902 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:53:45.472108 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:53:45.472209 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:53:45.472263 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:53:45.472419 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:53:45.472446 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
I20260928 21:53:45.474102 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 25691
I20260928 21:53:45.478860 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 25766
I20260928 21:53:45.483377 3075 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksdiuc7/build/release/bin/kudu with pid 25841
2026-09-28T21:53:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2699 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5408 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286139 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS